How do I set up alerting for LLM quality drops?

To set up alerting for LLM quality drops, treat it like observability for a production system: define what “quality” means, measure it continuously, compare to a baseline, and alert on statistically meaningful regressions.

1) Define the quality signals you care about

Pick a small set of metrics that reflect user value and model behavior. Common ones:

• Task success rate: did the model complete the task correctly?
• Human rating / CSAT: thumbs up/down, 1–5 scores
• Ground-truth accuracy: exact match, F1, pass/fail on labeled eval sets
• Hallucination rate: unsupported or fabricated claims
• Safety / policy violation rate
• Tool-use success rate: if the model calls APIs/tools
• Conversation resolution rate: did the user need to re-ask or escalate?
• Latency / timeout rate: quality often drops when latency rises

Choose metrics that are:

• aligned with product goals
• available automatically or through sampling
• stable enough to trend over time

2) Build an evaluation pipeline

You usually want two layers:

Online monitoring

Measure live traffic continuously:

• user feedback
• completion rate
• error rate
• token usage
• refusal rate
• tool-call success
• response length changes
• embedding similarity to reference answers (if useful)

Offline evals

Run a fixed benchmark set regularly:

• curated test prompts
• regression suites for high-value use cases
• adversarial / edge-case prompts
• new release comparison against prior model/prompt

This lets you catch drift before users do.

3) Establish a baseline

Alerting needs a reference point. For each metric, define:

• historical mean/median
• expected variance
• seasonal patterns by hour/day/week
• acceptable degradation threshold

Examples:

• “Alert if answer quality drops by >5% relative to 30-day baseline”
• “Alert if hallucination rate increases by >2 standard deviations”
• “Alert if pass rate on critical eval set falls below 92%”

Use rolling baselines rather than fixed thresholds when traffic is variable.

4) Use statistical alerting, not raw thresholds alone

LLM metrics are noisy. Better approaches:

• Rolling window comparisons: last 1h vs last 7d
• Control charts: Shewhart, EWMA, CUSUM
• Significance tests: proportion z-test, bootstrap confidence intervals
• Sequential monitoring: detect sustained degradation, not one-off noise

Example alert logic:

• Trigger if metric is below baseline by >3% for 3 consecutive windows
• AND sample size > N
• AND confidence interval excludes zero

5) Segment alerts by dimension

A global metric can hide localized failures. Break down by:

• model version
• prompt template version
• tenant/customer
• language
• region
• device/platform
• task type
• input length bucket
• tool availability
• retrieval enabled vs not

This helps you detect issues like:

• “quality only dropped for German prompts”
• “new prompt caused failures on long contexts”
• “one customer’s data format is breaking extraction”

6) Add release-aware monitoring

When you ship a new:

• model version
• prompt
• tool schema
• retriever index
• safety policy
• temperature/top-p setting

compare it directly against the previous version using:

• A/B tests
• shadow traffic
• canary rollout
• offline benchmark before full rollout

Alert on regression during canary before general release.

7) Instrument root-cause signals

Quality drops are easier to debug if you log relevant metadata:

• model ID
• prompt version
• temperature / top-p
• system prompt hash
• retrieval docs used
• tool calls and errors
• response confidence score if available
• refusal reason
• truncation / token limit hits
• latency and retry counts

Often a quality issue is really a retrieval or tool issue, not a model issue.

8) Sample and label

Not every output needs manual review. Use:

• random sampling
• risk-based sampling (long, low-confidence, safety-sensitive outputs)
• novelty-based sampling
• disagreement sampling between model versions

Then label a small set daily/weekly to validate automated metrics.

9) Choose alert thresholds carefully

Good alerting minimizes false positives and missed regressions.

A practical starting point:

• warning: 1–2% drop sustained over 3 windows
• critical: 5%+ drop or safety metric breach
• require minimum sample size
• use different thresholds for different traffic volumes

For low-volume endpoints, aggregate longer windows.

10) Integrate with your incident workflow

Alerts should go to:

• Slack / Teams
• PagerDuty / Opsgenie
• dashboards (Grafana, Datadog, Looker, etc.)

Include:

• metric name and current value
• baseline and delta
• affected segment
• model/prompt version
• recent deployment timestamp
• links to example bad outputs
• recommended owner/team

11) Track quality with a scorecard

A simple scorecard might include:

• accuracy/pass rate
• hallucination rate
• safety violations
• user thumbs-down rate
• latency p95
• tool success rate
• escalation rate

Display trend lines and alert thresholds.

12) Example alert strategy

For a customer-support LLM:

• Offline eval: 500 fixed test cases, run on every prompt/model change
• Online:
  • thumbs-down rate
  • resolution rate
  • escalation rate
  • refusal rate
  • average answer length
• Alerts:
  • thumbs-down rate increases by >20% vs 7-day baseline with p < 0.01
  • resolution rate drops below 85% for 30 min
  • safety violation count > 0 in critical intents
  • canary version underperforms control by >3%

13) If you want automation, add an LLM judge carefully

You can use another model to score outputs, but:

• calibrate against human labels
• keep a fixed judge prompt/version
• monitor judge drift
• don’t rely on it alone for critical alerts

14) A simple implementation pattern

1. Log every interaction with metadata
2. Compute metrics in 5–15 minute windows
3. Compare to rolling baseline
4. Segment by model/prompt/customer/task
5. Trigger alerts only when:
   • degradation is sustained
   • sample size is sufficient
   • confidence is high
6. Route to the owning team with examples

15) Recommended minimum setup

If you want a lightweight start:

• Track: thumbs up/down, task success, safety violations, latency, tool errors
• Compare against 7-day rolling baseline
• Alert on 3 consecutive bad windows
• Segment by model version and prompt version
• Keep a weekly human-reviewed eval set
• Canary every release

If you want, I can also give you:

1. a concrete alerting architecture,
2. Prometheus/Grafana-style metric examples,
3. SQL queries for LLM quality dashboards, or
4. a template alert policy for production.