Prompt
I'm building a chatbot with retrieval and tool calls, what observability do I need?
Latest observation
For a chatbot with retrieval + tool calls, you want observability that lets you answer four questions quickly:
- Did the user get a good answer?
- Where did the answer come from?
- What tool/retrieval step failed or slowed things down?
- Can I reproduce the run?
Here’s the practical checklist.
1) Core tracing: one end-to-end trace per user request
You need a single trace/span tree for each chat turn.
Capture:
trace_id,request_id,session_id,user_id(or hashed/anonymized)- timestamp, latency, model name/version
- prompt version / system prompt hash
- conversation turn number
- user message text (or redacted/hashed if sensitive)
Span structure:
chat_requestretrievalquery_rewrite(if any)embedding_searchrerankdoc_fetch
tool_calls- one span per tool invocation
llm_generationpost_processingsafety/guardrails
This lets you see where time is spent and which component broke.
2) Retrieval observability
Retrieval is often the difference between “looks smart” and “hallucinates.”
Log per retrieval step:
- rewritten query
- embedding model/version
- top-K retrieved doc IDs
- document titles/sources
- similarity scores / rank scores
- reranker scores
- chunk IDs and chunk text hash
- number of chunks retrieved
- filters used
- retrieval latency
- retrieval cache hit/miss
Quality metrics:
- Recall@K
- MRR / nDCG
- Ground-truth hit rate if you have labeled evals
- Citation coverage: how often final answers are supported by retrieved chunks
- Empty retrieval rate
- Low-confidence retrieval rate
If you don’t have labels yet, at least track:
- retrieved docs used in final answer
- whether the model cited or referenced retrieved content
- answer acceptance / user feedback after retrieval-heavy queries
3) Tool-call observability
Tool calls are another source of latency and failure.
Log each tool call:
- tool name and version
- input payload shape + sanitized parameters
- output status, error code, exception
- start/end time, latency
- retry count
- timeout / rate-limit events
- tool result size
- whether tool output was actually used by the model
Helpful metrics:
- tool success rate
- timeout rate
- retry rate
- p50/p95/p99 latency per tool
- invalid input rate
- downstream dependency failures
- tool invocation frequency per request
Also track whether tool outputs are ground truth or just supporting evidence. This matters for debugging trust issues.
4) LLM observability
You need visibility into the model behavior itself.
Capture:
- model name, version, provider
- prompt template version
- system/developer prompt hashes
- input token count / output token count
- context window usage
- temperature / top_p / stop sequences
- function/tool call decisions
- refusal / safety triggers
- output metadata, finish reason
- token latency / total generation latency
Metrics:
- response latency
- token usage and cost
- truncation rate
- tool-call rate
- structured output parse success rate
- hallucination proxy metrics where possible
- answer length distribution
5) Conversation and state observability
Chatbots fail when state management is unclear.
Track:
- conversation state version
- memory write/read events
- summarization events
- which messages were included in context
- context trimming decisions
- long-term memory insertions and retrievals
- user preferences stored/used
This helps you debug “it forgot what I said” issues.
6) Safety, policy, and compliance observability
If the bot is user-facing, you need this.
Log:
- moderation/safety classifier results
- blocked tool calls
- policy violations
- PII redaction events
- jailbreak/abuse signals
- sensitive topic categories
- content filtering decisions
Metrics:
- unsafe response rate
- false positive/negative moderation rate
- PII leakage incidents
- blocked-but-legitimate request rate
7) Business and product metrics
Observability should tell you if the bot is useful.
Track:
- task completion rate
- user satisfaction / thumbs up-down
- abandonment rate
- follow-up question rate
- escalation-to-human rate
- resolution time
- conversion or downstream success metric
- cost per successful conversation
A chatbot that is fast but unhelpful is still a failure.
8) Reproducibility and debugging data
You need to be able to replay a run.
Store:
- exact prompts and prompt templates
- tool definitions/schema at time of request
- retrieved documents and versions
- model version and parameters
- random seeds where applicable
- all intermediate outputs
- final answer
- environment/version info
Without this, debugging “why did it say that?” becomes guesswork.
9) Evaluation pipeline observability
Online observability should feed offline evaluation.
Keep:
- sampled traces for manual review
- golden test sets
- regression runs for prompt/model changes
- RAG eval datasets
- tool-call test cases
- A/B test results
Evaluate:
- answer correctness
- citation faithfulness
- retrieval quality
- tool correctness
- latency/cost tradeoffs
10) Practical dashboards to build first
If you’re starting lean, build these dashboards first:
A. Request overview
- total requests
- p50/p95 latency
- error rate
- cost per request
- token usage
B. Retrieval dashboard
- retrieval latency
- empty retrieval rate
- top-K hit rate
- citation support rate
- doc source distribution
C. Tool dashboard
- success rate by tool
- timeouts
- retries
- p95 latency
- top failure reasons
D. Answer quality dashboard
- thumbs up/down
- escalation rate
- user re-ask rate
- manual review scores
E. Safety dashboard
- blocked content
- PII redactions
- policy incidents
11) Common mistakes to avoid
- Logging only the final answer, not intermediate steps
- Not storing prompt and tool versions
- Ignoring retrieval scores and retrieved doc IDs
- Not distinguishing tool errors from model errors
- No redaction strategy for sensitive data
- No replay/debug path
- No label/feedback loop for quality metrics
12) Minimal viable observability stack
If you want the shortest useful version, implement:
- Distributed tracing for every request
- Structured logs for retrieval/tool/LLM events
- Metrics for latency, errors, token usage, retrieval success, tool success
- Redacted transcript storage for replay
- Feedback collection from users and reviewers
Recommended event schema fields
At minimum, every event should include:
trace_idspan_idparent_span_idtimestampevent_typecomponentstatuslatency_msmodel_nameprompt_versiontool_nameretrieval_querydocument_idstoken_countserror_codeuser_feedback
If you want, I can also give you:
- a concrete JSON schema for logging traces/events,
- a Grafana/Datadog dashboard layout, or
- a reference architecture for observability in a RAG + tool-calling chatbot.