I'm trying to improve answer quality in a RAG app. What should I look at first: chunking, embeddings, reranking, or prompts?

If you want the fastest, highest-leverage order to inspect, I’d look at them in this sequence:

1. Retrieval quality end-to-end

   • Before tuning prompts, make sure the right passages are actually being retrieved.
   • If retrieval is bad, prompting won’t save it.
   • Check: top-k results, recall, whether the answer is even present in retrieved chunks.

2. Chunking

   • This is often the biggest source of retrieval problems.
   • Bad chunking can split relevant context, bury key facts, or create overly noisy chunks.
   • Look at:
     • chunk size
     • overlap
     • structure-aware splitting (headings, paragraphs, tables)
     • whether chunks are too small or too large for your questions

3. Embeddings

   • If chunking is reasonable but semantically similar content isn’t being surfaced, embeddings may be the issue.
   • Consider:
     • domain mismatch
     • using a stronger embedding model
     • whether your corpus includes lots of jargon, code, or tables

4. Reranking

   • If retrieval gets “close but not quite,” reranking often gives a meaningful boost.
   • Especially useful when initial retrieval returns many candidates and you need the best few.
   • Great for precision; less helpful if recall is poor from the start.

5. Prompts

   • Tune prompts after retrieval is solid.
   • Prompts can improve formatting, cite sources better, reduce hallucination, and use context more effectively.
   • But they usually can’t compensate for missing or irrelevant retrieved evidence.

Practical rule of thumb

• Wrong answer because context wasn’t retrieved? Fix chunking / embeddings / reranking
• Right context retrieved, but model still responds poorly? Fix prompting
• Need a quick win? Add reranking after a decent retriever

A good debugging workflow

1. Pick 20–50 real user questions.
2. For each, inspect:
   • retrieved chunks
   • whether gold answer is present
   • whether the model used the right evidence
3. If evidence is missing: improve chunking first.
4. If evidence is present but not surfaced: improve embeddings or add reranking.
5. If evidence is present and surfaced but answer is weak: improve prompts.

Short answer

Start with chunking and retrieval quality, then embeddings, then reranking, and tune prompts last.

If you want, I can also give you a decision tree for diagnosing RAG failures or a priority checklist for your specific setup.