AI Glossary / Causal inference and experimentation / Intermediate

Pass at k (pass@k)

Definition

Pass at k, written pass@k, measures the chance or observed rate that at least one of k generated attempts solves a task under a specified correctness test. It is widely used to evaluate code-generation systems. A task counts as solved when one or more candidates pass the test; it does not require all k candidates to pass. The value of k is an essential part of the measure.

What it means

A generative model may produce several different answers to the same prompt. Pass@1 asks about success with one attempt. Pass@10 allows ten attempts and asks whether at least one succeeds. These evaluate different attempt budgets and should not be compared as though they describe the same experience.

The metric also assumes a way to decide whether a candidate passes. For code, this often means executing tests. Passing those tests is evidence under the benchmark's checks, not proof of correctness for every possible input.

Pass@k indicates whether a successful candidate is available among the attempts. It does not automatically show that a user or ranking system can identify that candidate without access to the evaluation answer or tests.

How it works

Define a set of tasks, a generation procedure, the number of attempts, and a pass criterion. For each task, generate candidates under the specified conditions and assess them. A direct evaluation with k attempts records success if at least one passes, then averages the success indicators across tasks.

Another common approach generates n candidates per task, where n is at least k, and counts c passing candidates. A standard estimator uses 1 − C(n−c,k)/C(n,k), where C(a,b) counts the ways to choose b items from a. It estimates success by considering subsets of k candidates rather than simply inserting an estimated single-attempt rate into a power formula.

Average the task-level estimates using the benchmark's stated weighting. Sampling assumptions, decoding settings, prompt format, and test coverage all affect interpretation. Record them alongside the score so comparisons remain meaningful.

Three practical examples

1. Coding benchmarks: researchers report pass@1 and pass@10 to distinguish single-attempt performance from the availability of a correct program after several generations.

2. Candidate-generation workflows: engineers assess whether generating several proposed data transformations supplies a valid option for a separate checker to select.

3. Budget comparison: a product team examines whether extra generated attempts improve task coverage enough to justify their additional computation and checking cost.

Think of it this way

Think of trying several keys on a lock and asking whether at least one opens it. More attempts can help, but owning a working key among many is different from knowing which key to choose first. The analogy is limited because generated answers may be highly similar, correctness tests may be incomplete, and attempts can have different computational costs.

Worked example

Suppose a fictional task has n = 5 generated candidates, of which c = 2 pass the tests. Estimate pass@2 by considering all two-candidate subsets. There are C(5,2) = 10 possible pairs. Three candidates fail, so C(3,2) = 3 pairs contain only failures.

Seven of the ten pairs therefore contain at least one passing candidate. The estimator gives 1 − 3/10 = 0.70. For the same pool, pass@1 is 2/5 = 0.40. These are task-level estimates, not percentages of lines of code that are correct.

For another task with one passing candidate out of five, pass@2 is 1 − C(4,2)/C(5,2) = 1 − 6/10 = 0.40. Giving the two tasks equal weight produces an estimated average pass@2 of (0.70 + 0.40)/2 = 0.55. The example illustrates the estimator; five samples and two tasks are far too limited to establish broad model quality.

Illustrative case study

Fictional case: Juniper Analytics evaluates an assistant that writes small parsing functions. A demonstration highlights a strong score after twenty candidates per task, but the proposed interface displays only one candidate.

The team therefore reports single-attempt results separately and evaluates a candidate-selection workflow if multiple attempts are to be used. It also adds tests for empty inputs and unusual separators, because the first test collection covered only typical cases. The case shows why the attempt budget, selection process, and checker matter. It does not imply any real model result or product improvement.

Common misconceptions and mistakes

Pass@k is not the fraction of the k candidates that pass. If one of ten attempts succeeds, that task is a success for direct pass@10 even though only 10% of its candidates passed.

Do not confuse k with the number of test cases inside a task. A program may face many tests, and the benchmark decides how those determine one candidate's pass status. Also avoid treating pass@100 as expected first-answer reliability.

The finite-sample estimator is not generally the same as 1 − (1 − c/n)^k. The original estimator was designed to avoid the bias of that plug-in calculation under its sampling model. Changing generation through feedback or adaptive retries creates a different protocol that must be described explicitly.

Business use

Pass@k helps businesses assess generation-plus-checking workflows, where several attempts may be useful if a reliable checker can identify acceptable output. It is less directly informative for a service that presents an unchecked first answer.

Compare systems at matched attempt and resource budgets, and measure selection success separately. Record timeouts, execution conditions, and how invalid outputs are counted. A score becomes decision-useful when it reflects the actual workflow and when the limits of the correctness test are understood.

People also ask

Can pass@k decrease as k increases?

Under a fixed generation distribution and pass criterion, the underlying at-least-one-success probability is nondecreasing. Different experimental settings can make reported comparisons misleading.

Does passing tests guarantee a correct program?

No. Tests can miss defects or omit parts of the intended specification.

Is pass@k only for code?

No. It can apply to tasks with a defined pass criterion, but that criterion and generation protocol must be explicit.

Check your understanding

Four attempts are generated for one task. Exactly one passes all required checks. How does that task contribute to a direct pass@4 result?

Answer: It contributes a success, or 1, because at least one attempt passes. It does not contribute 0.25; that would measure the fraction of passing candidates instead.

Related terms

• Benchmarks
• Sampling
• Decoding
• Code generation
• functional correctness
• benchmark
• test coverage
• pass@1
• candidate selection

Learn next

Make a five-candidate pool with two passes and enumerate its ten possible pairs. Verify the pass@2 calculation, then explain what extra component would be required to deliver the passing candidate reliably to a user.

More in Causal inference and experimentation

• Objective - An objective is a quantity that a learning or optimization process tries to improve, usually by minimizing or maximizing it.
• Out-of-bag evaluation - Out-of-bag evaluation estimates a bagged model's predictive performance by using, for each training example, only the component models that were trained...
• Pairwise metric - A pairwise metric is a rule used to compare two observations, commonly by calculating their distance or similarity.
• Permutation variable importances - Permutation variable importances measure how much a fitted model's performance changes when the values of an input variable are shuffled across evaluation...
• Perplexity - Perplexity measures how well a probabilistic model predicts an observed sequence, commonly a sequence of words or tokens.
• Pinball loss - Pinball loss is an asymmetric error measure used to train and evaluate quantile predictions.

Sources

Chen and colleagues, Evaluating Large Language Models Trained on Code, evaluation methodology and estimator: https://arxiv.org/pdf/2107.03374

The original paper grounds the pass@k definition and finite-sample estimator. The arithmetic, fictional workflow, and exercise are original; no current model ranking or performance claim is made.

Previous: Partitioning strategy Next: Passive-aggressive learning