Prompt
What should I use to compare models on quality and cost?
Latest observation
Oct 26, 2026
OpenAI APIWeb search: off
A good way to compare models on quality vs. cost is to evaluate them on the same tasks and then normalize by dollars per successful output.
Practical metrics to use
-
Task quality
- Human preference ratings
- Exact-match / pass-fail on your benchmark tasks
- LLM-as-judge scores, if used carefully
- Domain-specific metrics like accuracy, F1, BLEU, code tests passed, etc.
-
Cost
- Cost per 1,000 tokens or per request
- Better: cost per completed task or cost per successful task
- Include both input and output token costs
-
Efficiency
- Latency / time-to-first-token
- Throughput
- Reliability / failure rate
Best comparison approach
Use a table like:
| Model | Quality score | Avg cost/request | Success rate | Cost per success |
|---|---|---|---|---|
| A | 0.82 | $0.012 | 91% | $0.0132 |
| B | 0.85 | $0.030 | 96% | $0.0313 |
Then choose based on your target:
- If you want the best value: look for the lowest cost per success
- If you want the best raw quality: pick the highest quality score
- If you want the best tradeoff: compare Pareto frontier models
- A model is on the Pareto frontier if no other model is both cheaper and better
Simple rule of thumb
If you’re deciding quickly:
- Quality: benchmark on your real use cases
- Cost: compute total input + output token cost
- Decision metric: quality per dollar or cost per successful task
If you want, I can help you design a lightweight evaluation spreadsheet or benchmark template for your specific use case.