Why are my judge model scores not matching human ratings?

Judge model scores often fail to match human ratings because they’re usually optimizing for different signals than people are using. Common reasons:

1. Different criteria

   • Humans may judge helpfulness, correctness, nuance, tone, or trustworthiness all at once.
   • A judge model may overweight surface features like verbosity, fluency, or confident wording.

2. Prompt/spec mismatch

   • If the judge prompt is vague, the model may apply its own interpretation of “good.”
   • Small wording changes in the rubric can shift scores a lot.

3. Calibration issues

   • The judge’s score scale may not correspond to human score distributions.
   • It may compress scores, be too generous, or be too harsh.

4. Training bias / preference bias

   • The judge model may reflect biases from its training data, including preference for certain styles, templates, or lexical patterns.
   • It may prefer answers that “sound right” over answers that are actually correct.

5. Domain mismatch

   • If humans are rating a specialized domain, a general-purpose judge may not know the ground-truth subtleties.
   • It may miss edge cases or domain-specific conventions.

6. Insufficient context

   • Humans may use extra context not included in the judge input.
   • If the judge only sees the answer, but humans see the question, rubric, or conversation history, scores will diverge.

7. Low inter-rater agreement among humans

   • If humans don’t agree much with each other, matching them precisely is hard or impossible.
   • Your judge may actually be as consistent as humans, just not aligned to one annotator’s style.

8. Pairwise vs. absolute evaluation

   • Judges often work better at comparing two outputs than assigning absolute scores.
   • Human ratings are often absolute and subjective, which is harder to mimic.

9. Noisy labels

   • Human ratings can be inconsistent due to fatigue, anchoring, mood, or order effects.
   • The “true” target may be noisy, so the judge looks mismatched even when reasonable.

10. Distribution shift

• If the judge was validated on one set of prompts/answers and used on another, alignment can break.

What to do

• Tighten the rubric and make criteria explicit.
• Use pairwise comparisons instead of raw scores when possible.
• Calibrate judge outputs to human score distributions.
• Measure inter-annotator agreement first.
• Fine-tune or prompt the judge on examples from your exact task/domain.
• Evaluate whether the judge correlates better with human rankings than with absolute scores.

If you want, I can help you debug this systematically by looking at:

• your judge prompt,
• the human rubric,
• a few example outputs and scores,
• and the correlation metrics you’re seeing.