How do I evaluate classification outputs from an LLM?

Evaluating classification outputs from an LLM is similar to evaluating any classifier, but you should pay extra attention to label consistency, calibration, and prompt sensitivity.

1) Start with a clear task definition

Make sure you know:

• What the classes are
• Whether classes are mutually exclusive or multi-label
• What counts as correct for ambiguous cases
• How to handle abstentions / uncertain outputs

If the label space is unclear, metrics won’t mean much.

---

2) Build a labeled test set

Create a dataset with human-verified ground truth.

Good test sets should:

• Be representative of real inputs
• Include edge cases and hard examples
• Be large enough to estimate performance reliably
• Use consistent annotation guidelines

If possible, have multiple annotators and measure agreement.

---

3) Convert LLM outputs into class predictions

LLMs may output:

• A label directly
• A rationale + label
• A score or probability
• Free-form text

You need a deterministic mapping from output to prediction, e.g.:

• Exact label matching
• Regex/parsing rules
• “Best answer among options A/B/C”

For ambiguous outputs, define fallback rules:

• If no valid label, mark as invalid
• If uncertain, either use an abstain class or count as wrong, depending on the task

---

4) Use standard classification metrics

Common metrics:

Accuracy

• Overall fraction correct
• Useful when classes are balanced

Precision / Recall / F1

Especially important when classes are imbalanced.

For binary classification:

• Precision = of predicted positives, how many were correct?
• Recall = of actual positives, how many were found?
• F1 = harmonic mean of precision and recall

For multi-class:

• Compute macro, micro, and/or weighted averages:
  • Macro-F1: treats all classes equally
  • Micro-F1: aggregates all decisions; favors frequent classes
  • Weighted-F1: weighted by class support

Confusion matrix

Shows which classes are being confused with which others. Very useful for diagnosing model behavior.

---

5) Evaluate calibration if the LLM provides confidence

If the model outputs probabilities or confidence scores, measure whether they are meaningful.

Useful metrics:

• Expected Calibration Error (ECE)
• Brier score
• Reliability diagrams

A model can be accurate but poorly calibrated, which matters if you rely on confidence thresholds.

---

6) Measure robustness

LLM classification can vary with:

• Prompt wording
• Order of options
• Few-shot examples
• Temperature / decoding settings
• Input formatting

So evaluate across:

• Multiple prompts
• Multiple random seeds
• Multiple output formats

Report mean and variance if outputs are stochastic.

---

7) Check abstention and uncertainty handling

If your LLM can say “I’m not sure”:

• Measure coverage: how often it makes a prediction
• Measure selective accuracy: accuracy on answered cases only
• Plot accuracy vs coverage tradeoff

This is important in production settings.

---

8) Do error analysis

Don’t rely only on aggregate metrics. Inspect:

• Misclassified examples
• Systematic confusion between specific classes
• Failure cases by subgroup or input type
• Whether errors come from labeling ambiguity or model weakness

This often tells you how to improve the prompt or fine-tune.

---

9) Compare against baselines

Always compare the LLM to:

• Majority class baseline
• Simple rules or keyword baseline
• A smaller supervised classifier if available
• Previous model versions

If the LLM doesn’t beat simple baselines, it may not be worth using.

---

10) If the output is generated text, evaluate both form and meaning

For LLM-generated classification responses, check:

• Label validity: does it output only allowed labels?
• Format adherence: follows schema/instructions?
• Semantic correctness: chosen class matches the ground truth

A model can be semantically right but format-wrong, which matters in automation.

---

Practical recommendation

For most classification tasks, I’d recommend reporting:

• Accuracy
• Macro-F1
• Confusion matrix
• Per-class precision/recall/F1
• Invalid output rate
• If applicable: calibration metrics and coverage

---

Example evaluation workflow

1. Collect 200–1000 labeled examples.
2. Run the LLM with a fixed prompt and temperature 0.
3. Parse outputs into class labels.
4. Compute accuracy, macro-F1, confusion matrix.
5. Inspect 20–50 errors manually.
6. Repeat with prompt variants or seeds to test stability.

---

If you want, I can also give you:

• a Python evaluation script
• a template for evaluating multi-class vs multi-label classification
• or a checklist specifically for LLM prompt-based classification.