Can you recommend a retrieval evaluation tool for debugging failed searches in production?

Yes — if your goal is debugging failed searches in production, I’d recommend OpenSearch/Elasticsearch’s built-in Search Profiler plus a retrieval evaluation layer like Ragas / TruLens / LangSmith depending on your stack.

But if you want a single practical tool specifically for retrieval debugging, my top recommendation is:

Arize Phoenix

Why it’s good:

• Designed for LLM/RAG and retrieval observability
• Helps inspect:
  • retrieved documents
  • query embeddings
  • similarity scores
  • traces for failed lookups
• Good for production debugging
• Can compare retrieval results across requests and spot failure patterns

When it’s the best choice

Use Phoenix if you want to answer questions like:

• “Why didn’t the right passage get retrieved?”
• “Are we embedding the query incorrectly?”
• “Did reranking hurt recall?”
• “Which user queries are failing most often?”

---

Other strong options

LangSmith

Best if you already use LangChain/LangGraph.

• Excellent tracing
• Good debugging of retrieval chains
• Easy to inspect failed runs end-to-end

TruLens

Good for:

• retrieval quality evaluation
• feedback functions
• monitoring over time

Ragas

Best for:

• offline retrieval/RAG evaluation
• metrics like recall, faithfulness, context precision/recall
• not as strong for live production debugging as Phoenix/LangSmith

Elasticsearch/OpenSearch Profiler

Best if your retrieval is classic search over an index.

• Query execution breakdown
• Slow/failed query diagnosis
• Great for search-engine-level debugging, less for RAG-specific evaluation

---

My short recommendation by use case

• RAG/LLM retrieval debugging in production: Arize Phoenix
• LangChain-based apps: LangSmith
• Offline retrieval benchmarking: Ragas
• Traditional search engine troubleshooting: Elasticsearch/OpenSearch Profiler

If you tell me your stack — for example vector DB + reranker + LLM, or Elasticsearch/OpenSearch, or LangChain/LlamaIndex — I can recommend the best fit more precisely.