← back to the archiveCover illustration for “Pass@k: the chance at least one of k AI attempts works”
TERM6mo ago·Published · Updated ·by Andy Padia

Pass@k: the chance at least one of k AI attempts works

In short: Pass@k estimates the chance that at least one of k generated attempts is correct, which measures capability more than repeat-run reliability.

Pass@k is an evaluation metric that estimates the probability that at least one of k independent attempts by a model solves a task. OpenAI's Codex paper from July 2021 used it for its HumanEval benchmark, where each attempt is checked by unit tests. Pass@1 is single-try success; pass@10 lets the model try ten times and counts success if any one works.

Pass@k in plain words

It is like taking ten shots at a basketball hoop and counting it as a success if any one goes in. That is useful to know, but very different from asking whether the player scores reliably.

How it works

The Codex paper generates n samples per task, with n at least k (it used n = 200 and k up to 100), counts the c samples that pass, and computes an unbiased estimate of the chance that a random set of k includes at least one pass. Reported scores average this over all tasks. A higher k always gives a higher or equal number.

The gap can be wide. In that paper, Codex solved 28.8% of the HumanEval problems, and 70.2% when it was allowed 100 samples per problem.

Example

As an illustration, a model solves a task in 3 of 10 samples. Its pass@1 for that task is about 30%, while its pass@10 is 100%. For a user who gets one answer, 30% is the experience.

Where you'll meet it

  • Code-generation benchmarks such as HumanEval
  • Model papers and launch posts
  • Coding-agent evaluations that allow multiple attempts
  • Reliability research using pass^k, the chance that all k attempts succeed

What it changes for you

For buyers, check which k a vendor quotes: pass@1 reflects a single answer, while higher k assumes you can generate many and automatically pick the right one. For teams building agents that act without review, pass^k is the more honest reliability measure. The τ-bench paper from June 2024 proposed it and reported that a gpt-4o function-calling agent succeeded on fewer than half of its tasks, with pass^8 below 25% in the retail domain.

Pass@k vs pass^k

Pass@k asks "did any of k attempts succeed?" Pass^k asks "did all k attempts succeed?" The first measures capability; the second measures reliability.

Common mistake

My view: comparing a pass@10 score from one vendor with a pass@1 score from another. Always compare at the same k, and for production agents, look for repeat-run consistency before best-of-many.

Evals · SWE-bench · Benchmark contamination · Regression evals

From the archive: How to evaluate an LLM feature before production

Compare vendors at the same k, and judge production agents on repeat-run consistency.

#field-guide#evaluation#benchmarks#metrics#coding-agents

Sources

  1. arxiv.org/abs/2107.03374
  2. arxiv.org/abs/2406.12045
← older drop
Jailbreak: tricking an AI into breaking its own rules
newer drop →
Vibe coding: building software by prompt, not by reading

related drops

explore all 557 drops →
← back to the archiveday 175