Prompt
I'm building a product where we need to compare multiple LLMs on quality and latency
Latest observation
A good way to do this is to treat it like an evaluation platform, not just a benchmark script.
1) Define what “quality” means for your product
Different apps need different metrics. Common ones:
- Task success: did the model complete the user goal?
- Factual correctness: is the answer accurate?
- Instruction following: did it obey format, tone, constraints?
- Groundedness / citation quality: if using RAG, does it stick to sources?
- Safety / policy compliance: does it avoid disallowed outputs?
- User satisfaction proxy: pairwise preference or human rating
For many teams, the best starting point is:
- pairwise human preference
- plus a few task-specific automated checks
2) Measure latency in a way that matches production
Don’t just measure average response time. Track:
- TTFT (time to first token): very important for UX
- total completion time
- tokens/sec
- p50 / p95 / p99 latency
- error rate / timeout rate
Also control for:
- prompt length
- output length
- model context window
- streaming vs non-streaming
- retries / rate limits
- concurrency level
3) Build an eval set that reflects real usage
Use a mix of:
- real production prompts
- synthetic edge cases
- hard failure cases
- “gold” labeled examples
Make sure your dataset spans:
- short / long prompts
- easy / hard tasks
- common user intents
- multilingual if relevant
- structured output use cases if relevant
4) Compare models fairly
Keep variables constant:
- same prompt template
- same decoding settings where possible
- same tool access
- same retrieval context
- same evaluation rubric
If you’re testing chat models, include:
- system prompt
- conversation history
- tool-call scenarios
- streaming setup
5) Use a scoring approach
A practical setup is:
Automated metrics
- exact match / F1 for structured tasks
- JSON validity
- schema adherence
- citation overlap
- unit tests for code generation
LLM-as-judge
Useful for:
- open-ended writing
- summarization
- reasoning quality
- helpfulness
Best practice:
- use pairwise comparisons
- blind the model identity
- calibrate with human labels on a subset
Human review
Still important for:
- ambiguous tasks
- subjective quality
- safety
- product-specific standards
6) Plot quality vs latency
The most useful output is often a Pareto chart:
- x-axis: latency
- y-axis: quality
- each point: a model/configuration
This helps you find the best tradeoff instead of just “best model.”
7) Consider cost too
In practice you usually want a 3-way comparison:
- quality
- latency
- cost
A slightly better model may not be worth it if it’s 3x slower or 5x more expensive.
8) Recommended workflow
- Gather 100–500 representative prompts
- Run each model under identical settings
- Log:
- input tokens
- output tokens
- TTFT
- total latency
- errors
- Score outputs using:
- automated checks
- pairwise LLM/human judgments
- Aggregate results by task type
- Choose a deployment policy:
- one default model
- fallback model
- route easy vs hard requests to different models
9) Tools you can use
Depending on your stack:
- OpenAI Evals
- LangSmith
- Weights & Biases
- Helicone
- Arize Phoenix
- custom benchmark harness in Python
10) Practical recommendation
If you want the fastest path to something useful:
- start with a Python benchmark script
- log TTFT, total latency, token counts, errors
- use pairwise evals for quality
- visualize quality vs latency vs cost
If you want, I can help you design:
- a benchmark schema
- a scoring rubric
- a Python harness to compare multiple LLM APIs
- a dashboard format for results