
Pass@k: the chance at least one of k AI attempts works
In short: Pass@k estimates the chance that at least one of k generated attempts is correct, which measures capability more than repeat-run reliability.
Pass@k is an evaluation metric that estimates the probability that at least one of k independent attempts by a model solves a task. OpenAI's Codex paper from July 2021 used it for its HumanEval benchmark, where each attempt is checked by unit tests. Pass@1 is single-try success; pass@10 lets the model try ten times and counts success if any one works.
Pass@k in plain words
It is like taking ten shots at a basketball hoop and counting it as a success if any one goes in. That is useful to know, but very different from asking whether the player scores reliably.
How it works
The Codex paper generates n samples per task, with n at least k (it used n = 200 and k up to 100), counts the c samples that pass, and computes an unbiased estimate of the chance that a random set of k includes at least one pass. Reported scores average this over all tasks. A higher k always gives a higher or equal number.
The gap can be wide. In that paper, Codex solved 28.8% of the HumanEval problems, and 70.2% when it was allowed 100 samples per problem.
Example
As an illustration, a model solves a task in 3 of 10 samples. Its pass@1 for that task is about 30%, while its pass@10 is 100%. For a user who gets one answer, 30% is the experience.
Where you'll meet it
- Code-generation benchmarks such as HumanEval
- Model papers and launch posts
- Coding-agent evaluations that allow multiple attempts
- Reliability research using pass^k, the chance that all k attempts succeed
What it changes for you
For buyers, check which k a vendor quotes: pass@1 reflects a single answer, while higher k assumes you can generate many and automatically pick the right one. For teams building agents that act without review, pass^k is the more honest reliability measure. The τ-bench paper from June 2024 proposed it and reported that a gpt-4o function-calling agent succeeded on fewer than half of its tasks, with pass^8 below 25% in the retail domain.
Pass@k vs pass^k
Pass@k asks "did any of k attempts succeed?" Pass^k asks "did all k attempts succeed?" The first measures capability; the second measures reliability.
Common mistake
My view: comparing a pass@10 score from one vendor with a pass@1 score from another. Always compare at the same k, and for production agents, look for repeat-run consistency before best-of-many.
Related terms
Evals · SWE-bench · Benchmark contamination · Regression evals
From the archive: How to evaluate an LLM feature before production
Compare vendors at the same k, and judge production agents on repeat-run consistency.


