Jargonautica
Measuring Performance

LLM as a Judge

Also called: Model as a Judge, AI Judge

Using one language model to score the output of another. For open-ended work there is no single correct answer to check against, and paying people to read every result does not scale, so the practical compromise is to have a capable model grade against a written rubric. It works better than most people expect, and it has consistent biases you have to correct for.

Technical view

Known biases include position bias, favouring whichever answer is shown first, verbosity bias, favouring longer responses, and self-preference, favouring output from the same model family. Mitigations include randomising order and averaging, using pairwise comparison rather than absolute scores, providing detailed rubrics with worked examples, and calibrating against a human-labelled subset. Agreement with human judgement should be measured rather than assumed, and the judge treated as an instrument that requires its own validation.


Part of AI Terminology Index, section: Measuring Performance.

How it connects

LLM as a Judge links to 16 other terms, and the 12 closest are drawn here. The faint lines are links between those terms. See it in the full graph.

Benchmarks and EvalsBenchmarks and EvalsHuman Evaluation ProtocolHuman Evaluation Pr…Golden DatasetGolden DatasetBLEU and ROUGEBLEU and ROUGEArena and Elo RatingArena and Elo RatingRubricRubricMachine Learning OperationsMachine Learning Op…SycophancySycophancyInter-rater ReliabilityInter-rater Reliabi…SummarisationSummarisationReasoning Models and Test-Time ComputeReasoning Models an…Scalable OversightScalable OversightLLM as a Judge