Prompt
How do I make sure my knowledge base is actually being used?
Latest observation
To make sure your knowledge base is actually being used, you want to verify it at three levels: retrieval, generation, and business impact.
1) Confirm retrieval is happening
If your system uses RAG or search + LLM, first check whether relevant KB content is being fetched.
What to log
- Query text
- Retrieved documents/chunks
- Similarity scores / rank
- Whether the KB was consulted at all
- Final answer source citations, if supported
What to look for
- Relevant docs consistently appear in top-k results
- Retrieved chunks match the user’s intent
- The system isn’t falling back to general model knowledge only
Quick test Ask questions that are only answerable from the KB:
- “What is our refund policy for annual plans?”
- “What is the internal process for onboarding contractors?”
- “Which SLA applies to enterprise customers?”
If the answer is correct and cites or reflects KB content, that’s a good sign.
2) Add source attribution
Require the assistant to show where it got the answer from.
Options:
- Inline citations
- “Sources used” section
- Confidence + source list
- Quote snippets from retrieved passages
This helps you detect:
- Hallucinations
- Answers generated without KB support
- Outdated or irrelevant document use
If the assistant can answer without showing any source, it’s harder to prove the KB is being used.
3) Build a test set of KB-specific questions
Create a small evaluation set with:
- Common questions
- Edge cases
- Recently changed policies
- Ambiguous phrasing
- Questions with known answers in specific documents
For each question, define:
- Expected answer
- Required source document
- Acceptable paraphrases
Then run regression tests whenever you update:
- the KB
- chunking
- embeddings
- retrieval settings
- prompts
- model versions
4) Measure retrieval quality
Track metrics like:
- Recall@k: did the right document appear in top k?
- Precision@k: how many retrieved chunks were actually relevant?
- MRR: how highly ranked the correct doc was
- Answer grounding rate: how often answers are supported by retrieved text
If retrieval is weak, the model may still sound confident while using the wrong info.
5) Check user behavior and outcomes
A KB is “being used” only if it improves the workflow.
Track:
- Deflection rate: fewer escalations to humans
- Resolution rate
- First-contact answer accuracy
- User follow-up rate: do users have to ask again?
- Agent adoption: are support reps copying answers from the KB?
Also compare:
- With KB enabled vs disabled
- Before vs after KB updates
6) Look for failure modes
Common reasons the KB exists but isn’t being used:
- Retrieval threshold too high or too low
- Chunking breaks important context
- Queries aren’t being rewritten well
- The prompt tells the model to “answer from general knowledge”
- The KB is outdated or low quality
- The answer is in the KB but not in a retrievable form
- The system silently falls back when retrieval fails
7) Force a “KB required” mode for validation
For testing, temporarily configure the assistant to:
- answer only when supporting evidence is retrieved
- refuse to answer if no relevant KB passage is found
- surface “I couldn’t find this in the knowledge base”
This is useful to prove whether the KB is actually supplying answers.
8) Review real transcripts
Sample actual conversations and inspect:
- Was a KB document retrieved?
- Did the answer match the document?
- Did the model invent details?
- Did the user accept the answer?
A small manual review of transcripts often reveals issues that metrics miss.
9) Make the KB observable
If possible, build a dashboard with:
- Top queries
- Top retrieved docs
- No-result rate
- Citation rate
- Answer acceptance rate
- Most common unanswered questions
- Stale docs frequently retrieved
This makes KB usage visible instead of assumed.
Practical minimum checklist
If you want the simplest reliable setup, do these 5 things:
- Log retrieved documents for every query
- Require citations in answers
- Create a gold test set of KB-only questions
- Track recall@k and answer grounding
- Review a sample of live transcripts regularly
If you want, I can also help you design:
- a KB usage audit checklist
- a RAG evaluation harness
- or a dashboard schema for tracking whether the KB is being used.