How do I compare prompt versions for an LLM before shipping?

A good way to compare prompt versions before shipping is to treat prompts like code: define a target, build a test set, run both versions under the same conditions, and compare results with both automatic metrics and human review.

Practical workflow

1) Define what “better” means

Before testing, decide the main success criteria:

• Task success: does the model produce the right answer?
• Format compliance: valid JSON, correct schema, no extra text?
• Safety/compliance: fewer policy violations, fewer hallucinations?
• Tone/brand: more concise, more helpful, more consistent?
• Cost/latency: fewer tokens, faster responses?

Pick a small set of primary metrics so you don’t optimize everything at once.

---

2) Create a representative eval set

Build a dataset of real or realistic inputs:

• common cases
• edge cases
• failure cases
• adversarial or ambiguous cases

Include expected outputs if possible. Even 50–200 well-chosen examples can be very useful.

Tip: separate your data into:

• development set: for prompt iteration
• holdout set: only for final comparison

---

3) Run both prompt versions side by side

Use the exact same:

• model
• temperature / top_p
• system instructions
• tools / function schemas
• input set

For each test case, generate outputs from:

• Prompt A = current version
• Prompt B = candidate version

If the output is stochastic, run multiple samples per input and compare distributions, not just one response.

---

4) Score the outputs

Use a mix of scoring methods:

Automatic checks

• exact match / substring match
• structured output validity
• regex or schema validation
• unit-test style assertions
• retrieval precision/recall, if relevant

LLM-as-judge

Have a separate judge prompt compare A vs B on clear criteria:

• correctness
• completeness
• clarity
• safety
• style

Best practice: use blind A/B comparisons and randomized order.

Human review

For important prompts, manually inspect a sample of outputs, especially:

• borderline cases
• failures
• high-impact user flows

---

5) Track failure modes, not just averages

Averages can hide regressions. Look at:

• worst-case examples
• category-level performance
• new errors introduced by the prompt
• cases where B is worse than A even if average is better

A prompt that improves 90% of cases but breaks 10% of critical ones may not be shippable.

---

6) Compare cost and latency

A better prompt can be more expensive or slower. Measure:

• input/output token counts
• average and p95 latency
• tool-call frequency
• retry rates

Sometimes the best prompt is the one with the best tradeoff, not the highest raw score.

---

7) Use a gating process

Before shipping, set thresholds like:

• no regression on critical tests
• ≥ X% win rate over baseline
• 100% schema validity
• no increase in safety violations
• cost increase < Y%

If the candidate fails a gate, iterate before release.

---

Recommended comparison methods

Simple A/B table

For each input, record:

• prompt version
• response
• pass/fail
• notes

This is the fastest way to get started.

Pairwise preference eval

Ask reviewers or a judge model:

• Which output is better, A or B?
• Why?

This is often better than absolute scoring for subjective tasks.

Weighted scorecard

Assign weights to criteria:

• correctness 50%
• format 20%
• tone 10%
• safety 20%

Then compute a total score for each prompt version.

---

Common pitfalls

• Testing on too few examples
• Using only “happy path” inputs
• Changing prompt and model settings at the same time
• Relying only on LLM judges without human checks
• Ignoring failure rate and edge cases
• Shipping without a holdout eval set

---

A simple template you can use

For each prompt version, record:

• Prompt ID / version
• Model name
• Temperature/top_p
• Dataset version
• Success rate
• Schema validity
• Safety violations
• Avg tokens
• Avg latency
• Human preference win rate
• Notes / regressions

---

Best practice summary

If you want the shortest reliable approach:

1. Build a small but representative eval set.
2. Run old vs new prompts under identical settings.
3. Score with automated checks plus blind human or LLM pairwise review.
4. Focus on regressions and critical cases.
5. Gate release on explicit thresholds.

If you want, I can also give you:

• a prompt comparison spreadsheet template
• a Python script for A/B prompt evals
• or an LLM-judge rubric you can use right away.