Jargonautica
Measuring Performance

Arena and Elo Rating

Also called: Elo, Chatbot Arena

Ranking models by having people compare two anonymous answers to the same question and pick the better one, then aggregating those votes into a rating like the one used for chess. It captures overall preference well and is vulnerable to style: people reliably prefer longer, more confident, better formatted answers.

Technical view

Pairwise comparisons are aggregated with a Bradley-Terry or Elo-style model producing a scalar rating with confidence intervals. Advantages are open-ended prompts from real users and resistance to contamination. Weaknesses include length and formatting bias, self-selected voter population, sensitivity to prompt distribution, and susceptibility to gaming through targeted campaigns. Style-controlled variants attempt to isolate substance from presentation.


Part of AI Terminology Index, section: Measuring Performance.

How it connects

Arena and Elo Rating links to 5 other terms. The faint lines are links between those terms. See it in the full graph.

LLM as a JudgeLLM as a JudgeBenchmarks and EvalsBenchmarks and EvalsSycophancySycophancyHuman Evaluation ProtocolHuman Evaluation Pr…Standard BenchmarksStandard BenchmarksArena and Elo Rating