The 7 Biases Hiding in Your LLM Judge

Your judge is lying to you (a little)

LLM judges are the cheapest eval you can run — and the easiest to fool. Audit for these seven biases before trusting a single score:

1. Position bias — swap A/B order; scores should not move.
2. Verbosity bias — longer ≠ better. Length-normalize or penalize.
3. Self-preference — models rate their own outputs higher. Use a different judge family.
4. Format bias — markdown tables charm judges. Strip formatting for content scores.
5. Sycophancy — judges agree with confident tone. Test with confidently-wrong answers.
6. Rubric drift — vague rubrics rot. Version them like code.
7. Anchor bias — showing a reference answer anchors scores. Decide if that is what you want.

The fix is boring

Calibrate against 50 human-labeled examples, report agreement, re-check monthly. Boring, and it works — we teach the full harness in LLM-as-a-Judge.