Can the same prompt produce different outputs?

Yes. Sampling can choose different tokens even when the prompt and model are unchanged, although other system changes can also affect results.

Is greedy decoding a type of random sampling?

In ordinary usage, no. It deterministically selects a highest-probability next token, subject to tie handling and implementation details.

Does a low-probability sampled token mean an error occurred?

No. Low-probability outcomes are part of a distribution. Whether the resulting output is useful or correct is a separate evaluation question.

AI Glossary / Generative AI and language / Beginner

Sampling

Definition

Sampling means drawing an outcome according to a probability distribution. In language generation, it usually means selecting the next token probabilistically from the model's predicted distribution, or from a distribution modified by a decoding rule. Higher-probability tokens are more likely to be selected, but are not guaranteed to appear on any particular draw. This creates variation across generated outputs without requiring the model's parameters to change.

What it means

A language model may assign probability to several possible continuations. Sampling uses those probabilities to choose among them. It differs from greedy decoding, which selects a highest-probability next token at each step.

The term has many uses elsewhere, including selecting survey participants or drawing training examples. The supplied language-model source makes generation the focus here. The same basic probability idea applies, but the sampled object and purpose differ.

Random selection is not uniform selection unless all eligible outcomes have equal probability. A token with probability 0.8 is much more likely than one with probability 0.01, even though either may be chosen.

How it works

The model receives the current context and produces scores for possible next tokens. These are converted into a probability distribution. A sampling procedure draws one token, appends it to the context, and repeats using the updated context.

Generation ends when the applicable stopping condition is reached, such as an end token or a configured length limit. Each selected token can change the probabilities of later tokens, so generated positions are generally not independent draws from one fixed distribution.

Some decoding methods adjust the probabilities or restrict the candidate pool before sampling. Such changes affect the effective output distribution. For reproducibility, record the model, input, decoding configuration, and relevant randomness controls rather than describing a run only as “sampled”.

Three practical examples

1. A brainstorming tool samples several possible slogans, then has a person select and revise candidates that meet the brief.

2. A language-model researcher samples continuations to inspect repetition, fluency, and failure patterns that may not be obvious from an average prediction score.

3. A testing team generates repeated answers to the same question to measure output variability and identify whether important facts remain consistent across attempts.

Think of it this way

Think of a weighted draw in which some tickets occupy much more space than others. Larger shares are more likely to be selected, but the next draw is not determined solely by which share is largest.

The analogy has limits. In language generation, the ticket weights are recalculated after each selected token because the context has changed. Also, a token's sampling weight reflects the model and decoding procedure, not an independent guarantee that its resulting statement will be correct.

Worked example

Suppose a toy next-token distribution is “report” with probability 0.5, “invoice” with 0.3, and “schedule” with 0.2. Divide the unit interval into [0, 0.5), [0.5, 0.8), and [0.8, 1).

A uniform random draw of 0.63 selects “invoice”. A draw of 0.92 selects “schedule”. Greedy decoding would instead choose “report” for this distribution.

Across many independent draws from this unchanged toy distribution, the expected proportions are 50%, 30%, and 20%. Ten draws need not contain exactly five, three, and two occurrences. In a real generated sentence, selecting “invoice” changes the next context, so the following draw uses a new distribution rather than necessarily repeating this table.

Illustrative case study

Fictional case: Oak Ideas offers alternative names for internal workshops. Sampling gives the editors more variation than its initial deterministic setup, but some suggestions drift away from the intended topic.

The team evaluates both diversity and brief compliance. It keeps the prompt set fixed while comparing decoding settings and records all candidates, including poor ones, rather than showing only the selected favorites.

The final workflow combines generation with editorial review. More variation is useful for this task, but the team does not interpret it as a general improvement in correctness or apply the same configuration automatically to factual extraction.

Common misconceptions and mistakes

Sampling does not mean that the model chooses words without regard to context. The draw is governed by the conditional distribution and any decoding modifications.

It also does not guarantee originality, factual accuracy, or better reasoning. A sample can repeat familiar material or contain an error. Generating more candidates increases opportunities for selection, but also adds cost and requires a reliable selection procedure.

Another mistake is treating a fixed random seed as a universal reproducibility guarantee across all software and hardware environments. Exact replay depends on the complete implementation and execution conditions, not just the seed value.

Business use

Sampling is useful when a task benefits from alternatives or when teams want to investigate variability. It can support drafting, exploration, and repeated evaluation of a generative application.

Choose the generation procedure according to the task's requirements. A creative workflow may value diversity, while a structured extraction workflow may prioritize consistency and evidence. Measure the final system, including any filtering or selection stage. If users see only the best of several sampled outputs, report the number of attempts and selection cost when comparing it with a single-output baseline.

Check your understanding

A token has probability 0.7. Must it appear exactly seven times in ten draws from that distribution?

Answer: No. Seven is the expected count, not a requirement for every finite sample. Random variation can produce other counts even when the sampler is working correctly.

Related terms

Probability distribution
Decoding
Greedy decoding
Temperature
Top-k sampling
Response set
next-token prediction
nucleus sampling
random seed
diversity

Learn next

Use a three-outcome probability table and map it to intervals between zero and one. Practice converting several random numbers into outcomes. Then compare repeated generated outputs from one documented model configuration, checking both variation and task success. Keep changes to the prompt and decoding settings separate so the comparison remains interpretable.

Decoding

Sources

Sources: Jurafsky and Martin, Speech and Language Processing, N-gram Language Models, https://web.stanford.edu/~jurafsky/slp3/3.pdf — sampling section read for conditional generation and interval-based selection. Hugging Face Transformers, Generation strategies, https://huggingface.co/docs/transformers/main/en/generation_strategies — primary distinction between sampling and greedy decoding. The probability table and fictional workshop case are original illustrations, not observations from a production model.

Previous: Sampled softmax Next: Sampling bias