AI Glossary / Generative AI and language / Intermediate
Top-p sampling
Definition
Top-p sampling, also called nucleus sampling, generates text by sampling from the smallest leading set of next-token candidates whose cumulative probability reaches a chosen threshold p. The retained probabilities are normalized before the draw. Unlike a fixed top-k cutoff, the number of candidates can change at every step, adapting to how concentrated or spread out the model's current probability distribution is.
What it means
A model can be very decisive at one point in a sentence and consider many continuations plausible at another. Top-p sampling uses probability mass to determine the candidate pool rather than always keeping the same number of tokens.
If one token already carries most of the probability, a small set can reach the threshold. If probability is spread across many candidates, a larger set is needed. The term “nucleus” refers to this retained high-probability portion.
The p value is not an accuracy guarantee, a confidence level for the response, or a percentage of the vocabulary. It specifies a cutoff in the next-token distribution used by the decoding procedure.
How it works
Sort candidate tokens from highest to lowest probability. Starting at the top, accumulate their probabilities until the chosen threshold is reached or crossed. Retain that leading set, discard the remaining candidates, and normalize the retained probabilities to sum to one.
Sample the next token, append it to the context, and obtain a new distribution for the next generation step. Recompute the nucleus each time rather than retaining the first step's candidate list throughout the answer.
Temperature and other score-processing rules may change the distribution before the cutoff is applied. Implementations can also specify tie handling or a minimum number of retained tokens. Record these details when exact reproduction matters; the basic concept does not make every software implementation identical at boundaries.
Three practical examples
1. A creative-writing tool uses a probability-based candidate pool when producing alternative continuations from the same opening paragraph.
2. A model-evaluation team compares top-p and top-k on prompts with both obvious completions and many plausible continuations, measuring usable diversity rather than parameter values alone.
3. A drafting application logs its top-p setting with the model version, prompt format, and temperature so a later review can identify changes in generation conditions.
Think of it this way
Think of collecting the largest slices of a pie until the collection accounts for a target share of the whole. A pie dominated by one large slice needs few slices; a pie cut into many similar slices needs more.
The analogy only describes selecting probability mass. It does not say the retained tokens are true or suitable, and it does not describe complete answers. Each generation step has a newly calculated “pie” shaped by the evolving context.
Worked example
Suppose five next-token probabilities, already in descending order, are 0.50, 0.25, 0.15, 0.07, and 0.03. Choose p = 0.80. The first two sum to 0.75, which is insufficient. Including the third reaches 0.90, so the nucleus contains three tokens.
After normalization, their probabilities are 0.50/0.90 ≈ 0.5556, 0.25/0.90 ≈ 0.2778, and 0.15/0.90 ≈ 0.1667. The remaining two cannot be drawn under this filter.
At a later step, suppose the leading token alone has probability 0.85. Under the same idealized p = 0.80 rule, that single token forms the nucleus. This shows how the candidate count adapts. The retained mass can exceed p because tokens are included as whole candidates; the method does not split a token's probability to hit the threshold exactly.
Illustrative case study
Fictional case: Amber Drafts generates short exhibit descriptions. A fixed candidate-count setting sometimes admits weak continuations when the next phrase is fairly obvious, while overly narrow settings limit useful variation elsewhere.
The team tests top-p configurations using a fixed collection of exhibit notes. Reviewers check factual fidelity, readability, and the range of acceptable descriptions. They include unsuccessful generations in the results and record how much editing each candidate needs.
This fictional comparison illustrates task-based evaluation. It does not establish that top-p is universally superior or that a particular threshold transfers unchanged to another model.
Common misconceptions and mistakes
Do not interpret p = 0.9 as “the answer is 90% likely to be correct.” It concerns retained next-token probability mass, not factual reliability.
Do not assume it retains 90% of tokens. Depending on the distribution, the nucleus might contain very few candidates or a substantial portion of the vocabulary.
Avoid confusing truncation with choosing the top token. More than one retained candidate can still be sampled. Also avoid judging a decoding setup from a single polished example; a useful setting should be assessed over representative requests and repeated generations where variability matters.
Business use
Top-p sampling can help manage the balance between output variation and restriction to likely continuations. For a business application, the relevant outcome is the fraction of usable responses and the effort required to review or edit them.
Compare configurations under the same prompts and quality criteria. Record all relevant decoding settings, because changing temperature can change which candidates reach the top-p threshold. Revisit the evaluation after model updates rather than treating a threshold as a permanent product-wide constant.
People also ask
Why can the candidate count change?
The distribution changes with the context. A concentrated distribution reaches the threshold with fewer candidates than a broadly spread one.
Does p = 1 remove this filter?
In the usual formulation, it retains the full probability distribution. Other active filters or constraints may still restrict generation.
Is nucleus sampling a training method?
No. It is a decoding choice for selecting output tokens from a trained model's predictions.
Check your understanding
With p = 0.80, the first two tokens total 0.75 and the first three total 0.90. How many does the basic top-p rule retain?
Answer: Three. The leading set must reach or exceed the threshold, and two candidates do not supply enough cumulative probability.
Related terms
- Top-k sampling
- Temperature
- Decoding
- Greedy decoding
- Token
- Sampling
- Nucleus sampling
- probability mass
- response diversity
Learn next
Learn temperature sampling next to see how scaling can change the nucleus even when token ranking stays the same. Then compare fixed-count and probability-threshold filters on several toy distributions before evaluating their effect on complete responses.
More in Generative AI and language
- Token-mixing - Token-mixing is the exchange or combination of information across token positions inside a neural network.
- Tokenization - Tokenization is the process of turning text into smaller units called tokens so a language-processing system can work with it.
- Tokenizer - A tokenizer is the component that converts text into the token units and numerical identifiers expected by a language-processing system.
- Transformer - A Transformer is a neural-network architecture built around attention mechanisms that combine information from different positions in an input.
- Transformer Language Model - A Transformer Language Model is a language model that uses Transformer layers to build contextual token representations and make language predictions.
- Transformer Network - A Transformer Network is a neural network assembled from Transformer-style components, especially attention and feed-forward layers that build contextual...
Sources
Holtzman and colleagues, The Curious Case of Neural Text Degeneration: https://arxiv.org/abs/1904.09751
Hugging Face generation utilities, TopPLogitsWarper: https://huggingface.co/docs/transformers/main/en/internal/generation_utils
The original paper's abstract and primary implementation documentation were read. The numerical distributions, evaluation guidance, and Amber Drafts case are original teaching material. Historical research results are not presented as a universal performance guarantee.
Previous: Top-k sampling Next: Topic-specific PageRank