Why is my RAG app answering from the wrong docs?

When a RAG app answers from the “wrong” documents, it’s usually one of a few issues in the pipeline rather than the LLM itself.

Common causes

1) Retrieval is returning the wrong chunks

This is the most common cause.

Possible reasons:

• Bad chunking: chunks are too large, too small, or split mid-thought.
• Weak embeddings: the embedding model doesn’t capture your domain well.
• Poor metadata filtering: the retriever isn’t restricting by source, tenant, date, product, etc.
• Top-k too high: relevant chunks are present, but noisy chunks dominate the context.
• Similarity metric mismatch: cosine vs dot product vs normalized vectors can affect ranking.

2) The index is stale or mixed up

Your vector store may contain:

• old document versions,
• duplicates,
• wrong tenant/customer data,
• documents from multiple corpora without proper separation.

This often happens after re-indexing or partial updates.

3) Query rewriting is over-aggressive

If you use:

• query expansion,
• multi-query retrieval,
• self-querying retrievers,
• LLM-generated search terms,

the system may broaden the query too much and pull unrelated docs.

4) The reranker is not helping

If you use a reranker:

• it may be undertrained,
• configured badly,
• or reranking only a subset of noisy candidates.

A bad reranker can promote the wrong chunk to the top.

5) The prompt is letting the model “freewheel”

Even with correct retrieval, the LLM may:

• ignore context,
• use prior knowledge,
• hallucinate,
• answer from the most semantically similar but incorrect chunk.

If your prompt doesn’t strongly constrain the model to cite or ground answers in retrieved context, this can happen.

6) Context window issues

If too many chunks are stuffed into the prompt:

• relevant evidence can get buried,
• the model may attend to the wrong chunk,
• later chunks may overshadow earlier ones.

7) Ingestion / preprocessing errors

Documents may be wrong before retrieval even starts:

• OCR mistakes,
• bad text extraction,
• duplicated headers/footers,
• wrong language handling,
• HTML cleanup removing important content.

8) Multi-tenant / permissions bugs

If different users should only see certain docs, a permissions bug can make the retriever search across the wrong namespace or collection.

---

How to debug it quickly

Step 1: Inspect retrieved chunks

For a bad answer, log:

• the exact query,
• top-k retrieved chunks,
• similarity scores,
• metadata for each chunk.

Ask:

• Is the right doc even being retrieved?
• If yes, is it ranked too low?
• Is metadata filtering missing?

Step 2: Test retrieval without generation

Use retrieval only:

• show top 5 chunks for a query,
• manually verify whether they’re relevant.

If retrieval is wrong, the issue is upstream of the LLM.

Step 3: Compare chunking strategies

Try:

• smaller chunks with overlap,
• section-aware chunking,
• preserving titles/headings.

Often a chunk containing the answer is present, but the boundaries are bad.

Step 4: Check the index contents

Look for:

• duplicates,
• outdated docs,
• wrong sources,
• malformed metadata,
• mixed datasets.

Step 5: Tighten filters

If you have metadata like:

• document type,
• product line,
• customer,
• version,
• locale,

apply them aggressively.

Step 6: Reduce top-k and add reranking

Try:

• smaller retrieval candidate set,
• a reranker,
• thresholding by similarity score.

Step 7: Make the prompt stricter

Use instructions like:

• “Answer only from the provided context.”
• “If the answer is not in the context, say you don’t know.”
• “Cite the supporting chunk IDs.”

---

A practical debugging checklist

• Are the right documents indexed?
• Are duplicates or stale versions present?
• Are metadata filters applied?
• Is chunking preserving meaning?
• Are embeddings appropriate for the domain?
• Are top-k and reranking tuned?
• Does the prompt require grounding in context?
• Is the app mixing namespaces or tenants?

---

If you want the fastest likely fix

In order, I’d check:

1. retrieved chunks,
2. metadata filters / namespace isolation,
3. chunking,
4. prompt grounding,
5. reranking.

If you want, I can help you debug it step by step if you share:

• your retriever setup,
• chunk size / overlap,
• embedding model,
• vector DB,
• and one example query with retrieved results.