Prompt
What should I use to measure output quality, not just latency and errors?
Latest observation
Oct 25, 2026
OpenAI APIWeb search: off
To measure output quality beyond latency and errors, you usually want a mix of task-specific quality metrics and human judgment.
A good starting set:
- Accuracy / correctness — Is the output factually or logically right?
- Relevance — Does it actually answer the prompt or solve the task?
- Completeness — Does it cover all required parts?
- Consistency / faithfulness — Does it stay aligned with the source data or instructions?
- Precision / recall / F1 — Useful for classification, extraction, retrieval, or QA.
- Similarity scores — e.g. BLEU, ROUGE, BERTScore for text generation, though these are imperfect.
- Human ratings — Often the best way to assess usefulness, fluency, and overall satisfaction.
- Pairwise comparisons — Ask which of two outputs is better; often more reliable than absolute scoring.
- Task success rate — Did the user complete the intended goal?
- Hallucination rate / groundedness — For LLMs, how often does it make unsupported claims?
If you’re evaluating an LLM or generative system, a practical framework is:
- Define the task goal.
- Choose one or two objective metrics if possible.
- Add human evaluation for quality dimensions that are hard to automate.
- Track failure modes separately, like hallucinations, omissions, or policy violations.
If you want, I can suggest the best quality metrics for your specific use case, like:
- chatbots
- summarization
- code generation
- retrieval/search
- classification
- extraction