Prompt
why is my AI search not citing the right docs
Latest observation
Common reasons AI search cites the “wrong” docs:
-
Retrieval is finding semantically similar but not exact sources
Vector search often returns documents that “sound related,” even if they don’t support the answer well. -
Chunking is too coarse or too fine
- Too coarse: a chunk contains multiple topics, so the model cites a section that isn’t actually the relevant one.
- Too fine: the model loses surrounding context and grabs a misleading snippet.
-
Ranking is not strong enough
If you’re doing pure vector search, the top results may be relevant in topic but not in evidence. A reranker usually helps a lot. -
Metadata or citation mapping is broken
The model may be quoting the wrong document because:- chunk IDs don’t map correctly back to source docs
- page/section offsets are wrong
- deduplication merges similar chunks incorrectly
-
Prompting doesn’t force grounded citations
If the model is allowed to answer freely, it may cite whichever retrieved text is most convenient instead of the best support. -
The source docs themselves are ambiguous
If multiple docs say similar things, the model may choose the wrong one unless you constrain by recency, authority, product/version, or document type. -
Retrieval is missing the best doc entirely
If the right doc isn’t in the top-k, the model can only cite the wrong ones.
How to debug quickly:
- Inspect retrieved chunks before generation
Check whether the correct doc is actually being retrieved. - Compare chunk text to the citation
Make sure the cited span directly supports the claim. - Test with a known question-answer pair
See whether retrieval or generation is failing. - Add a reranker
This often fixes “right topic, wrong doc.” - Use stronger metadata filters
Restrict by product, date, version, language, or document type. - Log source IDs and offsets
Verify the citation pipeline from chunk → source document → displayed citation.
If you want, I can help you debug this systematically.
Send me:
- how your search is built (vector DB, reranker, hybrid search, etc.)
- how you chunk documents
- an example query
- the retrieved docs/chunks
- the citation shown vs. the correct citation
and I’ll help pinpoint where it’s going wrong.