Is beam search always exact?

No. A finite beam prunes candidates, so it generally approximates search over all possible sequences.

Can sampling produce different results from the same prompt?

Yes. Random selection can vary unless the relevant execution conditions are controlled.

Does decoding always operate one token at a time?

No. Different model families and tasks use different inference procedures; autoregressive token selection is one important case.

AI Glossary / Natural language and speech / Intermediate

Decoding

Definition

Decoding is the inference process that turns a model's scores or probabilities into a selected output sequence. In language and speech systems, it can mean choosing a transcription, translation or generated text through a search or sampling procedure. It is distinct from training the model that supplies those scores.

What it means

A language model can assign probabilities to possible next tokens without prescribing one unique way to choose among them. Greedy selection, beam search and sampling can produce different outputs from the same model and input.

The word decoder can also name a neural-network component or an operation that converts stored token identifiers back into text. Those meanings are related to output processing but should not be conflated with the decoding strategy used to choose a sequence.

In speech recognition, decoding selects words or characters supported by the audio and model. In open-ended generation, it may deliberately trade concentration on high-probability continuations for variation. The appropriate objective depends on the task; a more probable sequence is not automatically more truthful, useful or faithful to a source.

How it works

For an autoregressive model, output is built using probabilities conditioned on the input and the already generated prefix. Greedy decoding selects the highest-probability next token at each step. This is simple, but the best immediate choice need not lead to the highest-probability complete sequence.

Beam search retains several candidate prefixes and extends them, pruning according to a score. A finite beam is an approximation and can discard a prefix that would later become preferable. Length treatment and stopping rules affect the result, so beam width alone does not fully specify the method.

Sampling instead draws from a distribution over possible tokens. Temperature or restrictions such as top-k and nucleus selection change that distribution. Such settings affect diversity and concentration; they do not directly measure factual uncertainty. Every strategy also needs termination behavior, such as an end token or a maximum output length.

Three practical examples

A transcription system searches candidate word sequences to select an output consistent with the audio and its scoring model.
A translation system compares several candidate continuations rather than committing immediately to the highest-scoring first word.
A brainstorming tool samples alternative phrasings, while a structured-output workflow applies additional constraints to the sequences it permits.

Think of it this way

Think of planning a route through a branching map where each next step has a score. Taking the best-looking next road can be different from finding the best complete route; keeping several options costs more effort but preserves alternatives.

The analogy is limited because model probabilities are learned scores, not reliable measurements of real-world value. Even a perfectly optimized route through the model's choices can produce an unsuitable answer if the scoring model or objective is wrong for the task.

Worked example

Why the best first choice can lose

Consider a toy model that always emits exactly two tokens. At the first position, A has probability 0.7 and B has probability 0.3. After A, the next-token probabilities for X, Y and Z are 0.4, 0.35 and 0.25. After B, X has probability 0.99 and Y has probability 0.01.

Greedy decoding chooses A and then X, giving sequence probability 0.7 × 0.4 = 0.28. But B followed by X has probability 0.3 × 0.99 = 0.297, which is higher. The other A continuations score 0.245 and 0.175; B Y scores 0.003. All five sequence probabilities sum to one.

Keeping both first-position candidates would allow this tiny search to find B X. That does not prove a fixed-width beam is globally optimal in general. It only demonstrates the limitation of making each local choice independently.

Illustrative case study

A fictional translation team changes decoding settings to make outputs less repetitive. Reviewers notice that wording becomes more varied, but some names and numerical details become less consistent across repeated runs.

The team evaluates faithfulness and formatting alongside diversity and response time. It records model version, decoding parameters and stopping rules for each comparison. For fields requiring exact reproduction, it adds task-specific checks rather than assuming a low temperature guarantees correctness.

The lesson is that decoding settings alter how model predictions become outputs. They do not repair missing knowledge or establish factual reliability by themselves. The fictional scenario illustrates an evaluation need without claiming a universal relationship between one setting and translation quality.

Common misconceptions and mistakes

Greedy decoding finds the most probable complete sequence. Local choices can miss a better continuation.
A wider beam always improves the user's result. Search score and task quality are not identical, and computation costs change.
Temperature is a truthfulness control. It changes the selection distribution rather than verifying claims.
Decoding and training are the same process. Decoding uses the trained model to produce outputs.

Business use

Decoding choices affect reproducibility, variety, latency and adherence to output requirements. Treat them as part of the system specification when comparing models or investigating changed behavior. A result produced under one set of settings may not be directly comparable with another.

Evaluate the qualities the application actually needs. A brainstorming interface may welcome varied outputs, while data extraction needs stable fields and evidence checks. Record any constraints or postprocessing separately so users can distinguish a model preference from an enforced output rule. Even deterministic selection can produce a consistently incorrect answer.

Check your understanding

A first token has the largest probability, but its best continuation makes the full sequence less probable than another branch. What does this show?

Answer: Greedy next-token choice does not guarantee the highest-probability complete sequence. Sequence-level comparisons require considering later choices too.

Related terms

Inference
Greedy decoding
Beam search
Sampling
Temperature
Top-k sampling
nucleus sampling
autoregressive model
stopping criterion

Learn next

Trace a small branching probability tree and compare greedy selection with a search that retains alternatives. Then separate the objective being optimized from the practical quality measure used to judge the output.

Sources

Jurafsky and Martin, Speech and Language Processing, Chapters 7 and 16: https://web.stanford.edu/~jurafsky/slp3/7.pdf and https://web.stanford.edu/~jurafsky/slp3/16.pdf. Language-model generation and speech decoding. The two-token probability tree is an original, arithmetically checked example, not an empirical model result.

Previous: Decoder-only model Next: Decomposition