I’m unhappy with reading random samples to judge prompt quality

Totally fair — random samples are often a weak way to judge prompt quality because they’re noisy, selective, and easy to overfit your impressions to.

Better options depend on what you mean by “prompt quality,” but here are stronger evaluation approaches:

1. Define a rubric

Score prompts against specific criteria, like:

• follows instructions
• produces correct format
• minimizes hallucinations
• handles edge cases
• is robust across phrasing variations

This makes evaluation less subjective.

2. Use a fixed test set

Create a small but representative suite of inputs:

• normal cases
• tricky cases
• adversarial cases
• ambiguous cases

Then compare prompt versions on the same set.

3. Compare outputs side by side

For each test case, run multiple prompt variants and judge them blind if possible. This helps reduce bias from knowing which prompt produced which result.

4. Measure task-specific metrics

If applicable, use quantitative measures:

• exact match
• F1 / precision / recall
• JSON validity
• schema compliance
• human preference score

5. Stress test

Try:

• unusual wording
• missing context
• conflicting instructions
• long inputs
• noisy inputs

Good prompts stay stable under variation.

6. Use error analysis, not just examples

Instead of “looks good overall,” ask:

• what types of failures happen?
• how often?
• under what conditions?
• are failures systematic?

7. Build an evaluation harness

Automate prompt testing so each change gets checked against the same benchmark set. That’s usually much better than ad hoc sampling.

If you want, I can help you design a prompt evaluation framework for your specific use case — for example:

• chat prompts
• extraction / structured output
• coding prompts
• customer support
• agent/tool-use prompts