production monitoring for llm

Production monitoring for LLMs usually means tracking the model, the application around it, and the business/user outcomes so you can catch quality, cost, latency, safety, and reliability issues early.

What to monitor

1. System health

• Latency: p50, p95, p99 response times
• Error rate: timeouts, provider errors, retries
• Throughput: requests per second, queue depth
• Availability: uptime of model endpoint and surrounding services
• Token usage: input/output tokens per request

2. Cost

• Cost per request
• Cost per successful task
• Token spend by route, customer, feature, or prompt version
• Unusual spikes in usage

3. Quality

• Task success rate: did the output satisfy the user?
• Human ratings: thumbs up/down, review scores
• Automated evals: exact match, similarity, rubric-based scoring
• Hallucination indicators: unsupported claims, citation failures
• Instruction-following rate

4. Safety and policy

• Toxicity / unsafe content rate
• PII leakage detection
• Jailbreak / prompt injection attempts
• Policy violation rates
• Refusal rate and over-refusal rate

5. Data and drift

• Prompt distribution changes
• Input length drift
• Topic drift
• Output length drift
• Retrieval quality drift if using RAG
• Embedding / semantic drift over time

6. Product/business metrics

• Conversion rate
• User retention
• Deflection rate for support bots
• Resolution time
• Escalation rate
• Revenue or engagement impact

---

Typical monitoring setup

Logs

Store for each request:

• request id
• timestamp
• user/session id
• prompt/template version
• model name/version
• parameters like temperature/top-p
• retrieved documents if applicable
• response
• token counts
• latency
• errors
• human feedback
• safety flags

Metrics

Expose time-series metrics to dashboards and alerts:

• request count
• error count
• latency percentiles
• token spend
• refusal rate
• safety violation rate
• evaluation scores

Traces

For multi-step LLM apps, trace each step:

• retrieval
• reranking
• prompt construction
• model call
• post-processing
• tool calls

This helps identify where failures happen.

---

Alerts to set up

• p95 latency above threshold
• error rate above threshold
• cost spike above threshold
• sudden drop in quality score
• rise in unsafe content
• retrieval failure rate increase
• output format violations increase
• provider outage or rate-limit saturation

---

Best practices

• Version prompts and models explicitly
• Compare against baselines and previous versions
• Segment by customer, use case, language, and route
• Sample outputs for manual review
• Run offline evals on a fixed test set before deployment
• Use canary releases / A/B tests
• Keep a rollback path
• Redact sensitive data in logs
• Monitor both model metrics and product outcomes

---

Example dashboard sections

1. Reliability: latency, errors, uptime
2. Cost: token spend, cost per request
3. Quality: task success, human rating, eval score
4. Safety: policy violations, PII, jailbreaks
5. RAG quality: retrieval hit rate, citation accuracy
6. Business: conversion, retention, resolution rate

---

Common tools

• Observability: OpenTelemetry, Datadog, Grafana, Prometheus
• LLM-specific: LangSmith, Arize Phoenix, WhyLabs, HoneyHive, Helicone, TruLens, Braintrust
• Logging/analytics: BigQuery, Snowflake, Elastic, ClickHouse

If you want, I can also give you:

1. a production LLM monitoring checklist,
2. a dashboard metric schema, or
3. a sample architecture for monitoring an LLM app in production.