Do I need evals before putting an agent in front of customers?

Yes — you should have some evals before putting an agent in front of customers, even if they’re lightweight at first.

Why

An agent can fail in ways that are hard to predict:

• Wrong answers with confidence
• Policy/safety violations
• Tool misuse or bad side effects
• Hallucinated actions like claiming something was done when it wasn’t
• Prompt injection / data leakage
• Edge cases that only show up under real customer behavior

Evals give you a way to catch those before users do.

What “enough” looks like

You don’t need a huge, perfect eval suite to launch. For an early release, aim for:

1. A small golden set of representative customer scenarios
2. Adversarial cases for known failure modes
3. Tool/action tests if the agent can take real actions
4. Safety/policy checks
5. Regression tests for bugs you’ve already seen

Minimum recommendation

Before launch, make sure you can answer:

• What are the top 5–10 tasks the agent must do correctly?
• What are the top failure modes that would hurt customers or your business?
• Do you have tests that fail when those things break?

Practical approach

If you’re moving fast:

• Start with 10–50 high-value eval cases
• Run them in CI or before prompt/model changes
• Use human review for anything customer-facing and ambiguous
• Add cases continuously from real conversations and incidents

Rule of thumb

If the agent can:

• say things to customers
• take actions
• or touch customer data

then you should not launch without evals.

If you want, I can help you design a minimal pre-launch eval checklist for your specific agent.