AI Glossary / Generative AI and language / Intermediate

Fraction of successes

Definition

The fraction of successes is the number of trials meeting a defined success condition divided by the total number of eligible trials. It is an observed proportion between zero and one, often expressed as a percentage. In AI evaluation, a trial might be a question, generated answer, tool-use attempt, or complete task. The phrase is a general statistical description rather than a distinct language-model algorithm, and its meaning depends on the success rule and denominator.

What it means

A statement such as “the model succeeded 80% of the time” is incomplete without knowing what was attempted and what counted as success. Producing valid JSON, answering correctly, and completing a customer's objective are different events. A response may succeed under one definition and fail under another. The observed fraction describes the evaluated sample; it does not automatically establish the probability of success for every future request, user group, or operating condition.

How it works

Define an eligible trial and a pass condition before collecting results. Record each outcome as success or failure under that rule, while separately tracking missing or unjudgeable cases if needed. Sum the successes and divide by the eligible count. Report both counts with the rate. For statistical uncertainty, choose a method that matches the sampling design; ordinary binomial calculations assume a particular trial model and may be inappropriate for heavily correlated attempts. If several attempts belong to one task, decide whether the unit is an attempt or the overall task.

Three practical examples

• Structured output: measure the fraction of responses that pass a specified schema validator, with semantic correctness assessed separately.
• Task completion: count how often an assistant reaches a predefined acceptable outcome within an allowed number of actions.
• Confidence calibration: compare predicted success probabilities with observed success fractions among comparable predictions, rather than interpreting the model's confidence as proof of correctness.

Think of it this way

Think of counting completed deliveries out of all scheduled deliveries. The fraction is easy to calculate, but it changes meaning if canceled trips disappear from the denominator or late deliveries are quietly counted as on time. The analogy explains the importance of counting rules; AI tasks can have more ambiguous outcomes and correlated repeated attempts than this simple picture suggests.

Worked example

A fictional assistant is evaluated on 50 tasks, with up to two attempts allowed per task. It succeeds on the first attempt for 35 tasks. Of the remaining 15 tasks, it succeeds on 10 second attempts and fails on five. First-attempt task success is 35 / 50 = 70%. Task success within two attempts is 45 / 50 = 90%.
Across all executed attempts, there are 50 first attempts plus 15 second attempts, or 65 attempts. Exactly 45 of those attempts succeed, giving an attempt-level success fraction of 45 / 65, approximately 69.2%. All three calculations are correct, but they answer different questions. Reporting only 90% hides the additional attempts and their cost. Reporting only 69.2% hides the fact that most tasks eventually completed. Because second attempts occur only after a first failure, these 65 attempts are not a simple collection of interchangeable independent trials. The evaluation should retain task identifiers and distinguish first-attempt reliability, eventual completion, latency, and resource use. None of these numbers says whether the successful outcomes were equally valuable or whether the five failures were especially serious.

Illustrative case study

A fictional team tests an AI assistant that fills out internal forms. Its early dashboard labels every syntactically valid form a success. Reviewers discover that some valid forms contain the wrong customer identifier. The team creates separate measures for valid structure and correct completed work. It keeps timeout cases in the task-completion denominator and shows retries explicitly. This makes the results more useful for deciding where human review is needed, even when the headline rate is less flattering.

Common misconceptions and mistakes

A high fraction based on a tiny or unrepresentative sample can be misleading. Ten successes out of ten does not establish perfect reliability. Excluding difficult or failed trials after seeing the result inflates the rate. Averaging group percentages without considering group sizes can differ from the overall proportion. A success label is not necessarily a positive social or business outcome; it is a defined event. Repeated prompts, shared documents, and common users can also create dependencies that affect uncertainty estimates.

Business use

Specify success in terms that reflect the business objective and preserve the raw counts. Show important subgroups and failure types alongside the overall rate. Record retries, abstentions, time limits, and exclusions so comparisons remain fair. Use a stable evaluation set for development and a genuinely reserved assessment for major decisions. If outcomes have different costs or severity, add those measures rather than assuming every success and failure carries equal weight.

People also ask

Is the fraction of successes the same as accuracy?

It can be when success means a correct prediction on each evaluated item, but task success may involve a broader condition.

What if no trials were run?

The fraction is undefined because the denominator is zero; reporting zero percent would misleadingly imply observed failures.

Can two systems with the same fraction differ substantially?

Yes. They may fail on different cases, require different numbers of attempts, or incur different costs and consequences.

Check your understanding

A system completes nine of ten tasks after retries but only six on the first attempt. Which fraction should be reported?

Answer: Report 60% first-attempt completion and 90% eventual completion, with the retry allowance and costs. Choosing one without naming its event can mislead the reader.

Related terms

• Bernoulli trial
• Binomial distribution
• Accuracy
• Confidence interval
• Calibration
• Success rate
• sample proportion
• pass criterion
• retry
• task completion
• denominator

Learn next

Create a small evaluation sheet with a task identifier, pass rule, attempt number, and outcome. Calculate both attempt-level and task-level fractions, then explain why they differ. Study confidence intervals for proportions next, followed by sampling and dependence, so a simple percentage is accompanied by an honest account of what the evidence supports.

More in Generative AI and language

• Fast decay - Fast decay describes a quantity decreasing quickly over a specified sequence of steps or period of time.
• Fine-tuning - Fine-tuning is additional training that adapts a pretrained model to a particular task, domain, or desired behavior.
• Flash Attention - FlashAttention is an efficient way to compute attention in neural networks by reducing data movement between different levels of GPU memory.
• Function Calling with LLMs - Function Calling with LLMs is a way for a language model to request the use of defined software functions through structured outputs.
• Generated text - Generated text is written content produced by a computational system, such as a language model, in response to inputs or instructions.
• Generative grammar - Generative grammar is an approach to describing language through explicit principles or rules that specify possible linguistic structures.

Sources

NIST Dataplot, Binomial Proportion: https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/binoprop.htm NIST Dataplot, Exact Binomial Confidence Limits: https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/exacbici.htm Scikit-learn, calibration_curve: https://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html

Previous: Fractiles Next: Fractional cascading