I added new docs and now the assistant is hallucinating more. How do I debug grounding problems?

When new docs make the assistant hallucinate more, it’s usually a retrieval/grounding issue, not a model issue. The model is often faithfully answering from the wrong context, insufficient context, or conflicting docs.

Here’s a practical way to debug it.

1) First determine which grounding failure you have

Common failure modes:

• No retrieval: relevant docs aren’t being found at all.
• Wrong retrieval: semantically similar but incorrect docs are being retrieved.
• Bad chunking: the right doc is found, but the relevant passage is split apart or buried.
• Context overload: too many chunks; the model latches onto noisy or conflicting text.
• Conflicting sources: new docs disagree with old docs and there’s no priority rule.
• Outdated index: the assistant is querying stale embeddings or a partial index.
• Prompt mismatch: the system prompt encourages answering even when evidence is weak.

2) Reproduce with a small set of failing questions

Make a short test set of 10–20 queries:

• known-answer questions
• questions that fail after the new docs were added
• edge cases / ambiguous questions
• questions that require one specific source

For each query, capture:

• user question
• retrieved chunks
• chunk scores/ranks
• final answer
• citations if you have them

This gives you a “grounding trace.”

3) Inspect retrieval directly

For each failing query, ask:

• Was the correct document retrieved?
• Was the correct passage in the top-k?
• Was it ranked too low?
• Were irrelevant docs retrieved instead?

If the relevant text is not in top-k:

• adjust chunking
• improve metadata
• add synonyms / better indexing
• use hybrid search
• increase top-k temporarily for diagnosis

If it is in top-k but not used:

• context ordering may matter
• the model may need stronger instructions to quote/cite evidence
• relevant chunk may be too long or diluted

4) Check chunking quality

Bad chunking is a very common cause.

Look for:

• chunks that cut off definitions midway
• tables or lists split across chunks
• policies whose exceptions are in a different chunk
• chunks that are too large and semantically mixed

Good chunking usually means:

• coherent sections
• overlap between chunks
• preserving headings
• keeping tables/FAQ pairs together if possible

A useful test: if you read only the retrieved chunk, can you answer the question confidently?

5) Look for contradictions introduced by the new docs

New docs often create ambiguity.

Check whether:

• new docs supersede old docs
• there are multiple versions of the same policy
• different teams wrote conflicting guidance
• the assistant is mixing internal docs with public docs

Fix by adding:

• document versioning
• source priority
• “latest wins” rules
• metadata like effective_date, owner, doc_type

6) Verify the index is correct

It’s surprisingly common that the embedding/index pipeline is the real bug.

Check:

• are all new docs ingested?
• are embeddings generated for the latest version?
• did the index rebuild finish?
• are you querying the same corpus you think you are?
• are filters applied correctly?

A few quick checks:

• search by exact unique phrase from the new doc
• confirm doc IDs exist in the index
• compare raw text vs indexed text
• test with a manual search outside the assistant

7) Reduce noise in the prompt/context

If the assistant gets too much context, it may hallucinate by synthesis.

Try:

• fewer retrieved chunks
• stricter citation requirements
• instructions like “answer only from provided sources”
• explicit fallback: “if evidence is insufficient, say you don’t know”
• put the most relevant chunk first

A good grounding instruction is:

Use only the supplied context. If the context does not contain the answer, say so explicitly.

8) Add retrieval evaluation metrics

If you can instrument it, track:

• Recall@k: does the correct doc appear in top-k?
• MRR: how high is the correct doc ranked?
• Answer groundedness: is each claim supported by retrieved text?
• Faithfulness / citation precision: are citations actually relevant?
• Hallucination rate on your test set

This helps distinguish:

• retrieval problem vs generation problem

9) Use a “source-first” debugging workflow

For each bad answer:

1. inspect the answer
2. highlight each factual claim
3. map each claim to a source sentence
4. any claim without support is a grounding failure

This is often the fastest way to spot the issue:

• unsupported claim
• wrong source
• partial quote used too broadly
• overgeneralization from one doc to another

10) Practical fixes that usually help

Depending on what you find:

• Hybrid retrieval: combine keyword + semantic search
• Reranking: use a cross-encoder or reranker
• Metadata filters: doc type, product, version, date
• Chunk overlap: especially for procedural docs
• Source priority rules: official docs beat drafts
• Citations required: answer only if support exists
• Confidence gating: abstain when retrieval is weak

A simple debugging checklist

For a failing question, ask:

• Did the right doc retrieve?
• Did the right passage retrieve?
• Was it ranked high enough?
• Was the chunk clean and complete?
• Were there conflicting docs?
• Was the index fresh?
• Did the prompt force evidence-based answering?

If you want a fast next step

Send me:

• one hallucinated question
• the retrieved passages
• the assistant’s answer
• how your docs are chunked/indexed

I can help you pinpoint whether it’s retrieval, chunking, ranking, conflicts, or prompting.