For reducing hallucinations in a document Q&A bot, the retrieval setup that usually works best is:

Best-practice default

Hybrid retrieval + reranking + grounded answer generation

1) Use hybrid retrieval

Combine:

• Dense embeddings for semantic match
• Sparse retrieval (BM25 / keyword search) for exact terms, names, IDs, and rare phrases

Why this helps:

• Dense retrieval finds conceptually related passages.
• Sparse retrieval catches exact wording the model might otherwise miss.
• Together they reduce “missed evidence,” which is a common cause of hallucinations.

2) Retrieve more than you need, then rerank

Typical pattern:

• Retrieve top 20–50 chunks
• Use a cross-encoder reranker or LLM-based reranker
• Keep the top 3–8 chunks for generation

Why:

• Initial retrievers are good at recall, not precision.
• Reranking improves relevance a lot before the LLM sees the context.

3) Chunk carefully

Use chunks that are:

• Large enough to preserve meaning
• Small enough to be precise

Common starting point:

• 300–800 tokens per chunk
• 10–20% overlap

Also consider:

• Structural chunking by headings/sections
• Keeping tables, bullet lists, and FAQs intact
• Storing metadata: title, section, page, date, source

4) Use “answer only from context” generation

Prompt the model to:

• Answer only using retrieved passages
• Say “I don’t know” if the answer isn’t in context
• Quote or cite supporting passages

This is important because retrieval quality alone won’t prevent hallucinations if the generator is allowed to improvise.

5) Add confidence/coverage checks

Before answering, verify:

• Is there enough supporting evidence?
• Are top chunks consistent with each other?
• Is the query out-of-scope?

If evidence is weak:

• Ask a clarifying question
• Return “not found in the documents”
• Or broaden retrieval and try again

---

Practical setup that often performs well

If you want a strong default architecture:

1. Chunk documents structurally
2. Index with both BM25 and embeddings
3. Retrieve top 30 using hybrid search
4. Rerank to top 5
5. Generate answer with citations only from those 5
6. Refuse or abstain if evidence is insufficient

---

Extra tactics that help a lot

• Query rewriting: rewrite user questions into search-friendly queries
• Multi-hop retrieval: retrieve, inspect, then retrieve again if needed
• Metadata filtering: restrict by product, date, document type, tenant
• Deduplicate near-identical chunks: prevents overcounting one source
• Source freshness ranking: prefer newer docs when relevant
• Evaluation set: build questions with known answers and measure recall/faithfulness

---

If you want one sentence

The most reliable setup is hybrid retrieval (BM25 + embeddings), followed by reranking, then a strict grounded-generation prompt with abstention when evidence is weak.

If you want, I can also suggest:

• a recommended chunk size and overlap for your document type, or
• a reference architecture for a production Q&A bot.