Jargonautica
Measuring Performance

Pass@k

The chance that at least one of k attempts is correct. It is the standard way of scoring code generation, because a model that gets it right one time in ten is genuinely useful when you can run the tests and keep the one that works, and useless if you can only take one shot.

Technical view

Estimated unbiasedly from n samples with c correct rather than by literally generating k. Pass@1 measures single-shot reliability; higher k measures whether a correct solution is within reach given a verifier. The gap between them quantifies how much value a selection mechanism adds, which is directly relevant to agentic systems that generate, test and retry. It requires a reliable verifier, so weak test suites inflate the measure.


Part of AI Terminology Index, section: Measuring Performance.

How it connects

Pass@k links to 5 other terms. The faint lines are links between those terms. See it in the full graph.

Agentic CodingAgentic CodingBenchmarks and EvalsBenchmarks and EvalsReinforcement Learning with Verifiable RewardsReinforcement Learn…Code ModelCode ModelStandard BenchmarksStandard BenchmarksPass@k