Prompt
Our LangSmith evals keep passing but users still get bad answers. How do I fix that?
Latest observation
If LangSmith evals are passing but users still get bad answers, the usual problem is that your evals are measuring the wrong thing, missing real failures, or too easy to game. The fix is to align evals with actual user pain and make them more production-like.
Common reasons this happens
1. Your eval set is too narrow
You may be testing “happy path” examples, but users hit:
- weird phrasing
- long multi-turn conversations
- ambiguous requests
- edge cases
- unsupported domains
- tool failures / slow tools
- retrieval misses
Fix: Build an eval set from real production traces, especially:
- low-rated sessions
- user re-asks / corrections
- abandoned sessions
- escalations / handoffs
- hallucination reports
- failed tool calls
2. Your grading criteria don’t match user satisfaction
A chain can score well on things like:
- relevance
- fluency
- citation presence
- format compliance
…but still be wrong, unhelpful, or misleading.
Fix: Add evals for:
- correctness
- completeness
- actionability
- refusal quality
- groundedness
- instruction-following
- task success
If possible, use a rubric that asks: “Would the user consider this answer successful?”
3. You’re not testing end-to-end behavior
A retrieval chunker, retriever, reranker, prompt, model, and tool layer can each look fine individually while the full system fails.
Fix: Evaluate the full production chain:
- query rewriting
- retrieval quality
- tool selection
- tool output handling
- response synthesis
- final answer quality
4. The eval model is too weak or inconsistent
LLM-as-judge evals can miss nuanced errors or reward confident nonsense.
Fix:
- calibrate your judge on human-labeled examples
- compare judge scores vs. human scores
- use multiple graders or pairwise comparisons
- keep a “gold” set of hand-reviewed examples
5. Distribution shift
Your evals may not reflect current traffic:
- new products/features
- seasonal questions
- new user segments
- changed docs / KB
- prompt/model updates
Fix: Refresh the eval dataset regularly from recent traces.
A practical way to fix this
Step 1: Build a “real failure” dataset
Create a dataset of:
- bad production traces
- near-misses
- user complaints
- outputs that required correction
- cases where users rephrased the same question
Label them with:
- expected outcome
- failure type
- severity
- whether the answer is salvageable
Step 2: Add failure-oriented metrics
Instead of only average scores, track:
- pass rate on hard cases
- critical error rate
- hallucination rate
- retrieval miss rate
- tool failure recovery rate
- “user had to ask again” rate
Step 3: Use stricter rubrics
Make the grader fail answers that are:
- partially correct but materially misleading
- unsupported by sources
- missing crucial caveats
- overconfident when uncertain
- not directly answering the question
Step 4: Slice by scenario
Don’t just look at the overall score. Slice by:
- question type
- intent
- language
- length
- customer segment
- tool usage
- retrieval hit/miss
- conversation depth
A model can look great overall but fail badly on one slice.
Step 5: Put production traces into the loop
Use LangSmith to:
- capture real traces
- sample failures daily
- annotate them
- automatically add them to your eval set
- rerun evals on every prompt/model change
Step 6: Add human review for critical flows
For high-stakes or revenue-critical use cases, keep humans in the loop:
- create a weekly review queue
- inspect “passing but suspicious” outputs
- review low-confidence or low-groundedness cases
What to check in LangSmith specifically
- Are you evaluating against the right target?
- Are your reference answers actually representative?
- Are you using a judge rubric that rewards the wrong behaviors?
- Are you only testing a handful of curated examples?
- Are you including production traces and negative examples?
- Are you measuring end-to-end task success, not just format/coverage?
A good target state
You want evals that answer:
- Does this work on real user traffic?
- Does it fail in the same ways users complain about?
- Do our changes reduce those failures without creating new ones?
Short version
Your evals are likely good at testing the benchmark, not the product. Fix that by:
- using real production failures
- broadening coverage
- tightening grading criteria
- evaluating the full end-to-end system
- slicing by scenario
- validating against human judgment
If you want, I can help you design a LangSmith eval strategy for your specific app (RAG, agent, customer support, code assistant, etc.) and give you a concrete checklist and rubric.
Brands