Prompt

Our LangSmith evals keep passing but users still get bad answers. How do I fix that?

Artificial Intelligence / AI Developer Tools1 observationLast seen Nov 17, 2026

Latest observation

Nov 17, 2026OpenAI APIWeb search: off

If LangSmith evals are passing but users still get bad answers, the usual problem is that your evals are measuring the wrong thing, missing real failures, or too easy to game. The fix is to align evals with actual user pain and make them more production-like.

Common reasons this happens

1. Your eval set is too narrow

You may be testing “happy path” examples, but users hit:

  • weird phrasing
  • long multi-turn conversations
  • ambiguous requests
  • edge cases
  • unsupported domains
  • tool failures / slow tools
  • retrieval misses

Fix: Build an eval set from real production traces, especially:

  • low-rated sessions
  • user re-asks / corrections
  • abandoned sessions
  • escalations / handoffs
  • hallucination reports
  • failed tool calls

2. Your grading criteria don’t match user satisfaction

A chain can score well on things like:

  • relevance
  • fluency
  • citation presence
  • format compliance

…but still be wrong, unhelpful, or misleading.

Fix: Add evals for:

  • correctness
  • completeness
  • actionability
  • refusal quality
  • groundedness
  • instruction-following
  • task success

If possible, use a rubric that asks: “Would the user consider this answer successful?”

3. You’re not testing end-to-end behavior

A retrieval chunker, retriever, reranker, prompt, model, and tool layer can each look fine individually while the full system fails.

Fix: Evaluate the full production chain:

  • query rewriting
  • retrieval quality
  • tool selection
  • tool output handling
  • response synthesis
  • final answer quality

4. The eval model is too weak or inconsistent

LLM-as-judge evals can miss nuanced errors or reward confident nonsense.

Fix:

  • calibrate your judge on human-labeled examples
  • compare judge scores vs. human scores
  • use multiple graders or pairwise comparisons
  • keep a “gold” set of hand-reviewed examples

5. Distribution shift

Your evals may not reflect current traffic:

  • new products/features
  • seasonal questions
  • new user segments
  • changed docs / KB
  • prompt/model updates

Fix: Refresh the eval dataset regularly from recent traces.


A practical way to fix this

Step 1: Build a “real failure” dataset

Create a dataset of:

  • bad production traces
  • near-misses
  • user complaints
  • outputs that required correction
  • cases where users rephrased the same question

Label them with:

  • expected outcome
  • failure type
  • severity
  • whether the answer is salvageable

Step 2: Add failure-oriented metrics

Instead of only average scores, track:

  • pass rate on hard cases
  • critical error rate
  • hallucination rate
  • retrieval miss rate
  • tool failure recovery rate
  • “user had to ask again” rate

Step 3: Use stricter rubrics

Make the grader fail answers that are:

  • partially correct but materially misleading
  • unsupported by sources
  • missing crucial caveats
  • overconfident when uncertain
  • not directly answering the question

Step 4: Slice by scenario

Don’t just look at the overall score. Slice by:

  • question type
  • intent
  • language
  • length
  • customer segment
  • tool usage
  • retrieval hit/miss
  • conversation depth

A model can look great overall but fail badly on one slice.

Step 5: Put production traces into the loop

Use LangSmith to:

  • capture real traces
  • sample failures daily
  • annotate them
  • automatically add them to your eval set
  • rerun evals on every prompt/model change

Step 6: Add human review for critical flows

For high-stakes or revenue-critical use cases, keep humans in the loop:

  • create a weekly review queue
  • inspect “passing but suspicious” outputs
  • review low-confidence or low-groundedness cases

What to check in LangSmith specifically

  • Are you evaluating against the right target?
  • Are your reference answers actually representative?
  • Are you using a judge rubric that rewards the wrong behaviors?
  • Are you only testing a handful of curated examples?
  • Are you including production traces and negative examples?
  • Are you measuring end-to-end task success, not just format/coverage?

A good target state

You want evals that answer:

  1. Does this work on real user traffic?
  2. Does it fail in the same ways users complain about?
  3. Do our changes reduce those failures without creating new ones?

Short version

Your evals are likely good at testing the benchmark, not the product. Fix that by:

  • using real production failures
  • broadening coverage
  • tightening grading criteria
  • evaluating the full end-to-end system
  • slicing by scenario
  • validating against human judgment

If you want, I can help you design a LangSmith eval strategy for your specific app (RAG, agent, customer support, code assistant, etc.) and give you a concrete checklist and rubric.

Brands

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.