Prompt
How do I monitor RAG chunk relevance and citation quality?
Latest observation
To monitor RAG chunk relevance and citation quality, treat it like an evaluation/observability problem with two layers:
- Was the right context retrieved?
- Did the answer cite the right evidence accurately?
Here’s a practical framework.
1) Monitor chunk relevance
A. Retrieval metrics
Track these per query:
- Top-k hit rate
Did any of the retrieved chunks actually contain the answer? - MRR / nDCG / Recall@k
Useful if you have labeled relevance data. - Similarity score distribution
Watch for:- too many low-score chunks
- score collapse after embedding/model changes
- score inflation from duplicate/near-duplicate chunks
B. Query-to-chunk semantic relevance
For each retrieved chunk, compute:
- query–chunk embedding similarity
- optionally rerank with a cross-encoder / LLM judge for finer relevance
Flag chunks that are:
- top-ranked but semantically weak
- repetitive or off-topic
- too broad / too narrow relative to the query
C. Human/LLM relevance labels
Create a small evaluation set and label chunks as:
- relevant
- partially relevant
- irrelevant
Then track:
- % of queries where top-3 chunks are relevant
- average relevant chunk rank
- false positive retrieval rate
D. Chunk quality signals
Relevance often depends on chunking quality. Watch:
- chunk length distribution
- overlap between adjacent chunks
- duplicate chunks
- chunks with too much boilerplate
- chunks with no semantic completeness
2) Monitor citation quality
A citation is “good” if it is:
- grounded: the cited chunk supports the claim
- precise: cites the specific supporting evidence, not just vaguely related text
- complete: covers all important claims
- honest: doesn’t cite irrelevant or contradictory text
A. Citation precision
For each cited chunk, ask:
- Does the citation directly support the claim?
- Is the cited span specific enough?
- Is there a mismatch between the cited text and the answer statement?
Metrics:
- citation precision = proportion of citations that truly support the claim
- citation coverage = proportion of answer claims backed by at least one citation
B. Faithfulness / groundedness
Use an entailment-style check:
- premise = retrieved/cited chunk(s)
- hypothesis = answer sentence/claim
Score whether the claim is:
- entailed
- partially entailed
- not entailed
This can be done with:
- NLI models
- LLM-based judges
- rule-based checks for simple factual answers
C. Citation granularity
Track whether citations are:
- at the document level only (often too coarse)
- at the chunk level
- at the sentence/span level (best for precision)
Too-coarse citations often look “present” but don’t actually prove the claim.
D. Citation mismatch cases to flag
Look for:
- cited chunk mentions the same topic but not the actual fact
- answer uses numbers/dates/entities not in the cited text
- answer cites a chunk that is outdated or contradictory
- citation points to a chunk that was retrieved but not used in generation
3) Build an evaluation pipeline
A good monitoring pipeline looks like this:
For each query:
- Log the query
- Log retrieved chunks + scores
- Log final answer
- Log citations per sentence/claim
- Run automated checks:
- retrieval relevance
- citation entailment
- unsupported claim detection
- contradiction detection
- Aggregate metrics over time
4) Useful dashboards
Create dashboards for:
Retrieval
- Recall@k
- MRR
- average similarity score
- % queries with no relevant chunk in top-k
- retrieval latency
Citation
- citation precision
- groundedness score
- unsupported claim rate
- average citations per answer
- % answers with at least one bad citation
Drift / regression
- metrics by index version
- metrics by embedding model version
- metrics by query category
- metrics over time after content updates
5) Recommended alerts
Set alerts for:
- sudden drop in Recall@k
- spike in unsupported claims
- spike in low-confidence citations
- large changes in similarity score distributions
- many queries with zero relevant chunks retrieved
6) Best-practice implementation pattern
A simple but effective approach:
- Retriever eval: labeled relevance set + Recall@k/nDCG
- Generator eval: sentence-level groundedness checks
- Citation eval: claim-to-citation entailment
- Human review: sample 50–100 queries weekly for spot checks
7) If you want a lightweight starter setup
Start with these three metrics:
- Top-5 retrieval relevance rate
- Unsupported claim rate
- Citation precision
That gives you a fast signal on whether the retriever and citations are doing their jobs.
8) Example rubric for citation quality
Score each claim from 0–2:
- 2 = fully supported by citation
- 1 = partially supported / needs more context
- 0 = unsupported or incorrect citation
Then report:
- average citation score
- % claims scoring 2
- % claims scoring 0
If you want, I can also give you:
- a concrete evaluation schema for logging RAG traces,
- a SQL/dashboard metric list, or
- a Python/LLM-based evaluator for relevance and citation faithfulness.