AI Glossary / Generative AI and language / Intermediate
Top-k sampling
Definition
Top-k sampling is a text-generation method that limits each next-token draw to the k highest-scoring candidate tokens. Their probabilities are normalized within that restricted set, and one token is sampled. The process repeats with a newly calculated candidate set after each selection. It reduces the chance of drawing from the long tail of low-probability tokens while still allowing variation among retained candidates.
What it means
A language model may assign some probability to thousands of possible next tokens. Sampling from the entire distribution can occasionally select a very unlikely continuation. Top-k sampling removes candidates below a chosen rank cutoff before the draw.
The parameter k is a count of candidate tokens, not a number of words to generate or a number of complete answers. A setting of k = 20 concerns the candidates at one generation step; the answer can contain many more than 20 tokens.
The cutoff is based on rank rather than a fixed probability threshold. This means the retained set can account for very different amounts of the original probability mass at different steps.
How it works
At a generation step, obtain the model's next-token scores after the relevant score-processing steps. Identify the k highest-scoring candidates, suppress the others, and normalize the retained scores into a probability distribution. Draw a token from that distribution and append it to the context.
Repeat using the model's distribution for the updated context. The identities and weights of the retained tokens can change at every step. Temperature or other decoding controls may also be applied, so the complete configuration matters.
With k = 1 and an unambiguous highest score, the selection reduces to choosing that top token. Tie handling and minimum-candidate rules can differ by implementation, so exact boundary behavior should be checked when reproducing a system.
Three practical examples
1. A short-story generator restricts draws to a manageable set of likely continuations while still producing alternative wording across runs.
2. A developer compares several k values on a fixed evaluation set to see how the candidate cutoff affects coherence and useful diversity.
3. An experimentation tool records top-k alongside temperature and stopping settings, ensuring that a model comparison is not accidentally a comparison of different decoding procedures.
Think of it this way
Think of asking a reviewer to shortlist the highest-ranked proposals, then making a weighted draw from that shortlist. A low-ranked proposal cannot win once it has been excluded.
The analogy is limited because generation shortlists tokens, not completed proposals. The shortlist is rebuilt after every choice, and a sequence of locally likely tokens can still form a poor or inaccurate response. Ranking by model score is not the same as checking a claim against evidence.
Worked example
Suppose five next-token candidates have probabilities 0.40, 0.25, 0.20, 0.10, and 0.05. With k = 3, retain the first three, whose total probability is 0.85.
After normalization, their sampling probabilities become 0.40/0.85 ≈ 0.4706, 0.25/0.85 ≈ 0.2941, and 0.20/0.85 ≈ 0.2353. The other two have zero probability under this filtered draw.
With k = 2, the retained mass is 0.65 and the two probabilities become approximately 0.6154 and 0.3846. This tighter cutoff excludes the third candidate even though it originally had a substantial 20% probability. The arithmetic illustrates the tradeoff at one step, assuming no other filters or score adjustments. It does not establish which setting produces the best complete answer.
Illustrative case study
Fictional case: Orchard Stories generates practice passages for a reading club. Reviewers find that some unrestricted samples take abrupt turns unrelated to the assigned topic.
The team compares top-k configurations while holding the prompts and other settings fixed. Reviewers score whether passages stay on topic, meet the reading level, and provide sufficiently distinct examples. They count all sampled passages, including unusable ones.
This fictional evaluation makes the tradeoff measurable. It does not assume that the smallest shortlist is best, because too much restriction can reduce the variety the reading club needs.
Common misconceptions and mistakes
Do not confuse top-k sampling with top-k retrieval, where a search system returns the highest-ranked documents. Both use a rank cutoff, but they operate on different objects for different purposes.
Do not assume k specifies how many tokens the answer will contain. Output length is governed by separate stopping rules and limits.
Avoid interpreting retained tokens as guaranteed valid choices. They are likely under the model, not necessarily factual, safe, grammatical in a complete continuation, or suitable for the user. Also avoid comparing k values without accounting for temperature, filtering order, and other decoding settings.
Business use
Top-k sampling can be useful when a product needs several plausible variations but wants to restrict very unlikely local choices. Its practical value should be measured through representative output review rather than inferred from the parameter alone.
Track the number of usable samples, their diversity, and the effort needed to select or edit them. When migrating to another model, retest the chosen k: its probability distributions may be sharper or flatter, so the same candidate count can create a different effective restriction.
People also ask
Does top-k sampling always choose the highest-ranked token?
No. For k greater than one, sampling can choose another retained token according to the normalized probabilities.
Is top-k the same as top-p?
No. Top-k uses a candidate count, while top-p uses a cumulative probability threshold, allowing the number of retained candidates to vary.
Can temperature and top-k be combined?
Yes, in implementations that support both. Record the complete processing setup because the settings jointly affect the final sampling distribution.
Check your understanding
The top three candidates have total probability 0.85. After top-k filtering with k = 3, should their probabilities still sum to 0.85?
Answer: No. They are normalized to sum to 1 for the filtered draw. Their relative proportions are preserved under simple renormalization, while excluded candidates receive zero probability.
Related terms
- Sampling
- Temperature
- Temperature sampling
- Top-p sampling
- Greedy decoding
- Probability distribution
- Logit
- nucleus sampling
- decoding strategy
Learn next
Learn top-p sampling next to compare a probability-mass cutoff with a fixed candidate count. Then examine the same model distribution under different filters, calculating both the retained mass and the normalized probabilities before discussing output quality.
More in Generative AI and language
- Token-mixing - Token-mixing is the exchange or combination of information across token positions inside a neural network.
- Tokenization - Tokenization is the process of turning text into smaller units called tokens so a language-processing system can work with it.
- Tokenizer - A tokenizer is the component that converts text into the token units and numerical identifiers expected by a language-processing system.
- Transformer - A Transformer is a neural-network architecture built around attention mechanisms that combine information from different positions in an input.
- Transformer Language Model - A Transformer Language Model is a language model that uses Transformer layers to build contextual token representations and make language predictions.
- Transformer Network - A Transformer Network is a neural network assembled from Transformer-style components, especially attention and feed-forward layers that build contextual...
Sources
Hugging Face Transformers generation utilities, TopKLogitsWarper: https://huggingface.co/docs/transformers/main/en/internal/generation_utils
Hugging Face generation strategies: https://huggingface.co/docs/transformers/main/en/generation_strategies
Primary documentation was read for candidate restriction and sampling behavior. The five-candidate calculation, evaluation guidance, and Orchard Stories case are original teaching examples. Exact tie behavior and processing order remain implementation-specific.
Previous: Top-k accuracy Next: Top-p sampling