I'm trying to monitor a RAG system in production. What metrics should I track?

For a production RAG system, it helps to track metrics across the whole pipeline: retrieval, generation, system performance, and business/user outcomes. The exact set depends on your architecture, but here’s a practical checklist.

1) Retrieval quality metrics

These tell you whether the right context is being fetched.

• Recall@k: Did the correct/desired documents appear in the top k retrieved?
• Precision@k: How many of the top k retrieved docs were actually relevant?
• MRR / nDCG: Ranking quality of retrieved results.
• Hit rate: Percent of queries where at least one relevant document was retrieved.
• Context relevance score: How relevant the retrieved chunks are to the query.
• Context coverage / support rate: Whether the retrieved context contains enough evidence to answer.

If you have labels, these are ideal. If not, you can estimate with LLM judges or human review.

2) Generation quality metrics

These measure whether the model answers well using retrieved context.

• Answer accuracy / correctness
• Faithfulness / groundedness: Is the answer supported by retrieved context?
• Hallucination rate: How often the model invents unsupported facts.
• Answer completeness: Did it address all parts of the query?
• Citation accuracy: If you provide citations, do they actually support the claim?
• Refusal rate: How often the system says “I don’t know” or refuses appropriately.
• Toxicity / safety violations: If relevant to your domain.

Common RAG-specific metrics:

• Answer relevance
• Faithfulness
• Context utilization: Did the model actually use retrieved info rather than ignore it?

3) End-to-end task metrics

These measure whether the system solves the user’s problem.

• Task success rate
• Resolution rate
• User satisfaction score
• Escalation rate to human/support
• Follow-up rate: How often users need to ask again because the first answer was insufficient
• Time to resolution

4) Retrieval/generation pipeline health metrics

These are operational metrics you want for production monitoring.

• Latency
  • retrieval latency
  • reranking latency
  • generation latency
  • end-to-end latency
• Throughput / QPS
• Error rate
  • embedding failures
  • vector DB timeouts
  • LLM API errors
  • parser failures
• Timeout rate
• Token usage
  • prompt tokens
  • completion tokens
  • total tokens per request
• Cost per query
• Cache hit rate if caching is used
• Fallback rate: how often the system uses backup logic/models

5) Index and corpus health metrics

Important if your knowledge base changes often.

• Document freshness / staleness
• Indexing lag
• Embedding drift: whether new embeddings differ significantly from old distributions
• Chunk size distribution
• Duplicate content rate
• Broken document rate
• Coverage of source corpus: how much content is indexed vs missing

6) Query and usage metrics

These help you understand demand and failure modes.

• Query volume by type/topic
• Query length
• Ambiguous query rate
• Out-of-domain query rate
• No-answer rate
• Top failure intents/topics
• User segment performance if relevant

7) Monitoring for drift and quality regressions

Track these over time, especially after changes.

• Performance by cohort
• Retrieval quality drift
• Answer quality drift
• Embedding/model version comparisons
• A/B test results for retriever or prompt changes
• Regression alerts when quality drops after deployment

8) If you can only track a small set

Start with these:

1. End-to-end latency
2. Error rate
3. Recall@k or hit rate
4. Faithfulness / groundedness
5. Answer correctness
6. Hallucination rate
7. Cost per query
8. User satisfaction / task success

9) A simple production dashboard layout

A useful dashboard usually has:

• System health: latency, errors, timeouts, throughput
• Retrieval quality: recall@k, hit rate, context relevance
• Answer quality: faithfulness, correctness, hallucination rate
• Business metrics: task success, satisfaction, escalation
• Drift/regression: trends by model version and time

10) Practical tip

For RAG, the most important distinction is:

• Is the retriever failing?
• Is the generator failing despite good retrieval?
• Is the user’s task failing even if the answer looks good?

That separation makes debugging much easier.

If you want, I can also give you:

1. a minimal metric set for a startup,
2. a production monitoring schema with event/log fields, or
3. a dashboard template for RAG observability.