What should I use for monitoring prompt and retrieval failures?

Use separate monitoring for prompt issues and retrieval issues, but connect them in one trace so you can see the full request path.

What to monitor

1) Prompt failures

Track things like:

• malformed prompt templates
• missing variables
• prompt drift after edits
• unsafe or unexpected model outputs
• low-quality responses tied to a specific prompt version

Helpful signals:

• prompt version
• model version
• input/output tokens
• latency
• error rate
• response quality scores or human feedback

2) Retrieval failures

Track:

• no documents retrieved
• irrelevant documents retrieved
• stale documents
• ranking issues
• embedding/vector search errors
• source document availability

Helpful signals:

• query text
• top-k results
• retrieval latency
• document IDs and scores
• recall/precision-like evals
• “no-hit” rate

Best practice

Use:

• application metrics for counts/latency/errors
• structured logs for prompt text, retrieved docs, and exceptions
• distributed tracing to correlate retrieval → prompt → generation
• evaluation/observability tooling for quality regressions over time

Good tooling options

Depending on your stack:

• OpenTelemetry for traces/metrics/logs
• Prometheus + Grafana for metrics dashboards
• Datadog / New Relic / Honeycomb for full observability
• LLM-specific tools like:
  • LangSmith
  • Arize Phoenix
  • WhyLabs
  • Weights & Biases for experiments/evals

Recommended setup

If you want a practical starting point:

1. Add request tracing across retrieval and prompt generation
2. Log prompt version, retrieved doc IDs, scores, and model output
3. Create alerts for:
   • retrieval no-hit rate spikes
   • prompt template errors
   • sudden quality drops
   • latency or token-cost spikes
4. Run periodic evals on a fixed test set to catch regressions

If you want, I can also recommend a specific monitoring stack for your environment (e.g. Python/LangChain, OpenAI API, Azure, AWS, etc.).