What should I use to catch hallucinations and prompt regressions before release?

To catch hallucinations and prompt regressions before release, use a combination of:

1. Goldens / eval sets

Create a fixed set of representative prompts with expected outputs or acceptance criteria.

• Best for: regression testing across prompt/model changes
• Include edge cases, ambiguous prompts, and known failure modes

2. Automated LLM evals

Run the model against those goldens and score for:

• factuality / hallucination rate
• instruction following
• format compliance
• refusal behavior
• citation correctness if applicable

Tools commonly used:

• OpenAI Evals / eval harnesses
• LangSmith evals
• Ragas for RAG systems
• DeepEval
• Promptfoo
• Helicone / Braintrust / Arize Phoenix for experiment tracking and evals

3. Human review

Have a small review loop for high-risk outputs.

• Best for: nuanced failures evals miss
• Especially important for customer-facing or regulated use cases

4. Red teaming

Actively probe for:

• hallucinated facts
• unsafe instructions
• jailbreaks
• prompt injection
• tool misuse

5. Production monitoring

After release, log and monitor:

• user complaints
• low-confidence outputs
• citation mismatches
• schema/format errors
• drift in response quality

Practical recommendation

If you want a simple starting stack:

• Promptfoo or DeepEval for automated regression tests
• A curated golden dataset
• A lightweight human review pass for critical cases

If you want, I can also suggest a specific evaluation workflow for:

• chatbots
• RAG apps
• tool-using agents
• customer support assistants