Can you show me how to tell whether a bad answer came from the prompt, the retriever, or the model?

Yes — the easiest way is to debug it as a pipeline:

1. Did the prompt tell the model the right thing?
2. Did retrieval bring back the right evidence?
3. Did the model fail to use that evidence correctly?

A practical way to tell is to inspect each stage separately.

---

1) Check whether the prompt is the problem

Look at the actual prompt sent to the model, including:

• system instructions
• user message
• retrieved context
• formatting/instructions for the answer

Ask:

• Is the task clearly stated?
• Are there conflicting instructions?
• Did the prompt say to answer only from context, or use general knowledge?
• Is the output format causing the issue?

Signs the prompt is the cause

• The model gives the wrong style, format, or scope even though the context is good.
• The prompt is ambiguous or contradictory.
• The prompt asks for something impossible or underspecified.
• The retrieved context is correct, but the model ignores it because the prompt didn’t emphasize using it.

Quick test

Try a minimal prompt with the same context:

• “Answer using only the context below.”
• If the answer improves, the original prompt was likely the issue.

---

2) Check whether the retriever is the problem

Inspect the documents/chunks returned by retrieval.

Ask:

• Did retrieval return the right documents?
• Were the most relevant chunks included?
• Was the important information split across chunks?
• Was the context too long, causing key evidence to be truncated?
• Did ranking put irrelevant chunks above relevant ones?

Signs the retriever is the cause

• The model’s answer is wrong because the needed evidence never appeared in the context.
• Retrieved chunks are off-topic, outdated, or incomplete.
• The answer would have been easy if the right passage had been retrieved.
• The model hallucinated because it had no supporting evidence.

Quick test

Manually provide the ideal evidence snippet:

• If the model answers correctly when you inject the right passage, retrieval was the problem.
• If it still fails, the issue is likely the prompt or model.

---

3) Check whether the model is the problem

If the prompt is clear and retrieval is good, the model itself may be:

• failing to reason over the evidence
• overconfidently hallucinating
• ignoring instructions
• not following the required output format

Signs the model is the cause

• Good context is present, and the prompt is clear, but the answer is still wrong.
• The model misreads the evidence or draws a bad inference.
• It answers correctly sometimes and fails inconsistently.
• It paraphrases the context but misses the key conclusion.

Quick test

Use the same prompt and context with:

• a stronger model
• a different decoding setting, e.g. lower temperature
• a “quote the exact supporting sentence” instruction

If performance improves a lot, the issue is likely model behavior.

---

A simple diagnosis workflow

Here’s a reliable sequence:

Step A: Freeze the inputs

Save:

• user query
• prompt template
• retrieved chunks
• model output

Step B: Test with gold context

Replace retrieved context with the exact passage that contains the answer.

• If it works now → retriever problem.
• If it still fails → model or prompt problem.

Step C: Simplify the prompt

Use a very direct prompt:

Answer only from the provided context. If the answer is not in the context, say “I don’t know.”

• If it improves → prompt issue.
• If not → model issue.

Step D: Compare against a stronger model

Run the same prompt and context through a better model.

• If the stronger model succeeds → model capacity/reasoning issue.
• If both fail → prompt or evidence quality issue.

---

Useful debug questions

For each bad answer, ask:

Prompt

• Was the instruction unambiguous?
• Did I ask for too much?
• Did I define the output format clearly?
• Were there conflicting instructions?

Retriever

• Did the right document get retrieved at all?
• Was the answer split across chunks?
• Did I retrieve enough context?
• Were the chunks ranked well?

Model

• Did it ignore obvious evidence?
• Did it hallucinate unsupported facts?
• Did it follow instructions partially but not fully?
• Did a stronger model do better?

---

A tiny decision tree

Bad answer →

• No relevant evidence in retrieved context?

  • Yes → retriever
  • No →

• Prompt unclear / conflicting / too restrictive?

  • Yes → prompt
  • No →

• Model misread or ignored good evidence?

  • Yes → model

---

Best practice: make each stage observable

Log these separately:

• query
• prompt template version
• retrieved docs and scores
• final context sent to the model
• model output

That way you can tell exactly where the failure happened.

---

If you want, I can also give you:

1. a debug checklist you can apply to every failure, or
2. a small evaluation script/pseudocode for diagnosing prompt vs retriever vs model errors.