Prompt
How do I root-cause a drop in answer quality after an index update?
Latest observation
To root-cause a drop in answer quality after an index update, treat it like a regression investigation across data, retrieval, ranking, and generation. A good approach is to isolate where the quality changed and compare “before vs after” systematically.
1) Confirm the regression is real
- Compare offline eval metrics before and after the index update:
- answer relevance / correctness
- retrieval recall / precision
- citation accuracy if applicable
- latency and timeout rates
- Use the same query set and the same evaluation harness if possible.
- Check whether the drop is:
- global, or only for certain query types
- limited to specific languages, domains, or documents
- correlated with traffic spikes or freshness windows
2) Determine where the failure is happening
Break the pipeline into stages and inspect each one:
A. Indexing / ingestion
Look for:
- missing documents
- duplicate documents
- stale documents removed or overwritten
- chunking changes
- metadata loss
- embedding model changes
- parsing/OCR failures
- field mapping changes
Useful checks:
- document count by source/type
- chunk count distribution
- embedding vector norms / anomalies
- ingestion error logs
- document freshness and update lag
B. Retrieval
Compare top-k results before and after for the same queries:
- Did the right documents stop appearing in top-k?
- Did irrelevant docs start dominating?
- Did scores shift unexpectedly?
Useful metrics:
- Recall@k
- MRR / nDCG
- top-1 / top-3 hit rate
- query coverage by source/domain
- score distributions
C. Re-ranking / filtering
If you use a reranker or post-filter:
- verify the reranker model/version
- inspect thresholds and filters
- look for metadata filters becoming too strict
- ensure reranker features are populated correctly
D. Generation
If retrieval is fine but answers worsened:
- prompt/template changes
- context truncation
- token budget issues
- model/version changes
- longer contexts causing important evidence to be cut off
- citation formatting errors
3) Diff the index state
Run a structured diff between old and new index snapshots:
- added/removed documents
- changed chunks
- changed metadata
- changed embeddings
- changed field mappings
- changed language/domain proportions
Look especially for:
- large increases in near-duplicate chunks
- chunks that became too small or too large
- important fields no longer indexed
- source-specific losses
4) Inspect failing queries manually
Take a sample of queries where quality dropped and compare:
- user query
- retrieved documents before vs after
- reranked order
- final prompt/context
- generated answer
For each failure, classify the cause:
- no relevant docs retrieved
- relevant docs retrieved but not ranked highly enough
- relevant evidence present but truncated
- model ignored evidence
- answer grounded in wrong document
- hallucination due to missing context
5) Segment by cohort
Identify whether the regression is concentrated in:
- short vs long queries
- navigational vs exploratory queries
- fresh content vs archived content
- specific document sources
- rare entities / long-tail queries
- multilingual traffic
This often reveals the culprit faster than looking at aggregate metrics.
6) Check operational changes
Sometimes the “index update” is not the true cause. Also inspect:
- ranking config changes
- synonym/boost rules
- hybrid search weights
- vector normalization
- ANN parameters
- cache invalidation
- shard rebalancing
- rate limits / timeouts
- fallback behavior
7) Reproduce with a canary or rollback
- If possible, route a small slice of traffic to the old index.
- Compare answer quality directly on live traffic.
- Roll back the index update if the regression is severe and isolated.
8) Build a regression report
For each failing query, record:
- query text
- expected answer / gold label
- old answer vs new answer
- old top-k vs new top-k
- which document(s) changed
- suspected root cause
This makes it easier to fix and prevents recurrence.
Practical debugging order
If you want the fastest path, do this in order:
- Check retrieval hit rate on a fixed eval set
- Diff top-k results before/after
- Inspect ingestion/index diffs
- Check reranker and filtering
- Check prompt/context truncation
- Review model/version/config changes
If you want, I can also give you a step-by-step RCA checklist, a table of metrics to compare before vs after, or a debugging template for RAG systems.