Prompt
LLM tracing and evals
Latest observation
LLM tracing and evals are the two most important pieces of making LLM apps reliable in practice.
What they are
Tracing
Tracing is the process of recording what happened during an LLM request end-to-end:
- user input
- prompts
- model calls
- tool calls
- retrieved documents
- intermediate outputs
- latency
- token usage
- errors
- final response
It gives you observability into the full execution path of an LLM app, especially if it uses:
- multi-step chains
- agents
- RAG
- tool/function calls
- multiple model providers
Evals
Evals are systematic tests that measure how well your LLM system performs on a set of examples or tasks.
They answer questions like:
- Is the answer correct?
- Is the output grounded in retrieved context?
- Did the agent use the right tool?
- Is the response safe and compliant?
- Did a prompt change improve quality or regress it?
Why they matter
Without tracing:
- failures are hard to debug
- prompt changes are risky
- you can’t see where latency comes from
- tool/RAG issues are opaque
Without evals:
- you have no objective way to compare versions
- improvements are based on vibes
- regressions slip into production
- you can’t benchmark model or prompt changes
Together:
- tracing helps you understand behavior
- evals help you measure quality
Common tracing data to capture
A good trace usually includes:
- request metadata: user/session ID, timestamp, app version
- inputs: raw user prompt, system prompt, chat history
- model info: model name, temperature, max tokens
- retrieval: query, top-k docs, scores, source IDs
- tool calls: tool name, args, outputs, failures
- intermediate steps: chain outputs, agent reasoning artifacts if appropriate
- performance: latency per step, total latency, tokens in/out
- outcome: final answer, user feedback, success/failure
Types of evals
1) Offline evals
Run on a fixed dataset before deploying changes.
Examples:
- Q&A accuracy
- code generation correctness
- summarization quality
- RAG faithfulness
- classification F1
2) Online evals
Run in production or near-production.
Examples:
- user thumbs-up/down
- task completion rate
- hallucination reports
- fallback rate
- conversion/support resolution metrics
3) Human evals
People judge outputs using rubrics.
Good for:
- nuanced quality
- helpfulness
- tone
- safety
- preference comparisons
4) Automated evals
Use programmatic checks or another model as judge.
Good for:
- exact match
- JSON validity
- citation presence
- groundedness
- policy compliance
- rubric scoring
A practical workflow
-
Trace everything
- instrument your app
- log prompts, tool calls, retrieval, outputs
-
Build a gold set
- collect real user examples
- label expected outputs or criteria
-
Define metrics
- correctness
- groundedness
- latency
- cost
- safety
-
Run evals on every change
- prompt changes
- model swaps
- retrieval changes
- tool changes
-
Compare versions
- baseline vs candidate
- track regressions and improvements
-
Deploy with monitoring
- production traces
- alert on anomalies
- sample outputs for review
Useful metrics
Depending on the task:
- Exact match
- Pass/fail
- Precision/recall/F1
- Human preference win rate
- Groundedness / faithfulness
- Hallucination rate
- Task success rate
- Latency
- Token usage / cost
- Tool success rate
- Retrieval hit rate
Tools people commonly use
Tracing/observability:
- LangSmith
- OpenTelemetry
- Helicone
- Arize Phoenix
- Weights & Biases Weave
- Datadog / Grafana custom instrumentation
Evals:
- LangSmith evals
- OpenAI evals
- TruLens
- RAGAS
- DeepEval
- Phoenix evals
- custom Python harnesses
Best practices
- Treat prompts like code: version them
- Use representative datasets, not just toy examples
- Include hard edge cases
- Separate retrieval quality from generation quality
- Keep evals lightweight enough to run often
- Review false positives/negatives in automated evals
- Track both quality and cost/latency
- Re-run evals whenever prompts, tools, models, or retrievers change
A simple example
For a support chatbot:
Trace
- user question
- retrieved KB articles
- model response
- whether a ticket was created
- latency and tokens
Eval
- Did it answer from the KB?
- Was the answer accurate?
- Did it escalate correctly when needed?
- Was the response polite and concise?
If you want, I can also give you:
- a tracing/evals architecture diagram,
- a Python example using LangSmith/OpenTelemetry, or
- a checklist for evaluating a RAG app.