Does greedy decoding find the best complete answer?

Not necessarily. It selects local next-token winners, and “best” also depends on a task-specific quality standard.

Is it the same as sampling?

No. Ordinary greedy decoding selects the highest-scoring candidate rather than drawing randomly.

Can constraints be used with it?

Yes. A decoder can greedily select among permitted candidates, but constraints change the available choices and do not automatically validate meaning.

AI Glossary / Natural language and speech / Intermediate

Greedy decoding

Definition

Greedy decoding is a method for generating a sequence by choosing the highest-scoring available next token at each step. In an autoregressive language model, this usually means selecting the token with the largest probability given the prompt and tokens already generated. Each choice becomes part of the context for the next choice; earlier choices are not reconsidered.

What it means

A language model can assign probabilities to many possible continuations. Greedy decoding turns those possibilities into one output by repeatedly taking the current local winner. This makes the selection rule easy to understand, but a sequence assembled from local winners need not be the most probable complete sequence.

The word greedy describes this decision strategy, not the model's motives or attitude. The same trained model can produce different outputs under different decoding methods. Changing the decoder changes how predictions are selected; it does not by itself retrain the model or improve its underlying knowledge.

How it works

First, the model processes the prompt and produces scores for the next token. Applicable constraints may exclude some candidates, and score transformations may alter the ranking. The decoder selects the highest-scoring remaining candidate and appends it to the context. Generation repeats until a stopping condition is reached, such as an end token or length limit.

For an unconstrained model distribution, selecting the largest probability is equivalent to selecting the largest corresponding logit. A tie needs a defined tie-breaking rule. Practical reproducibility also depends on the implementation and numerical conditions, so a greedy selection policy alone is not a universal guarantee of identical results across every system.

Unlike beam search, ordinary greedy decoding does not keep several competing partial sequences. Unlike probabilistic sampling, it does not draw the next token randomly from the distribution. These differences concern search and selection, not whether an answer is factually correct.

Three practical examples

1. Translation: A translator chooses each next target-language token greedily. A locally attractive word may constrain later phrasing, so reviewers compare whole-sentence meaning rather than only the next-token confidence.

2. Structured responses: A developer combines a greedy choice rule with permitted-token constraints for a small output format. The constraints help control form, while separate checks validate the supplied values.

3. Model comparison: An evaluation team fixes the decoding policy when comparing two models. Otherwise, a difference in generated answers could reflect selection settings as well as model capability.

Think of it this way

Imagine choosing a route by taking the shortest available next road segment at every junction. That is a local rule; it may lead to a longer total journey because the next junction matters too. The analogy illustrates why immediate preference need not optimize the whole path. Language-model probabilities are conditional scores, however, not physical distances, and generation does not usually have a fully explored road map.

Worked example

Consider an invented model that must produce exactly two tokens, followed by a certain end marker. At the first step, token A has probability 0.55 and B has probability 0.45. After A, the second-token probabilities are x: 0.51 and y: 0.49. After B, they are x: 0.95 and y: 0.05.

Greedy decoding chooses A because 0.55 exceeds 0.45, then x because 0.51 exceeds 0.49. The resulting sequence Ax has probability 0.55 × 0.51 = 0.2805. Yet Bx has probability 0.45 × 0.95 = 0.4275, which is higher.

The other complete sequences have probabilities Ay = 0.2695 and By = 0.0225. All four probabilities sum to one. Thus Bx is the most probable complete sequence in this toy model, while greedy decoding returns Ax. Equal sequence lengths and the certain end marker avoid introducing a length-comparison complication. Higher sequence probability would still not establish truth or usefulness in a real application.

Illustrative case study

Fictional case: A small software team uses a language model to draft short catalog descriptions. It initially assumes that always choosing the most likely next token will produce the best overall description.

The team reviews outputs for missing attributes, unsupported claims, repetition, and readability. It compares decoding policies on the same held-out products and retains human review for publication. Greedy decoding remains a useful baseline because its selection rule is simple, but the team stops treating that rule as a quality guarantee. This example illustrates an evaluation process, not a measured improvement from a real company.

Common misconceptions and mistakes

A common mistake is equating the most likely next token with the most likely entire answer. The worked example shows why that inference fails. Another is assuming that a high-probability sequence must be correct: probability reflects the model's predictions, not independent verification.

Greedy decoding is also not simply a synonym for every interface setting described as “deterministic.” Some products combine temperature settings with other transformations or restrictions. Read the actual decoder specification, including tie handling and stopping rules, before drawing conclusions about behavior.

Business use

For business use, greedy decoding provides a clear baseline for evaluating generation systems. Record the model, prompt, constraints, stopping conditions, and decoder configuration alongside results. Check task outcomes such as factual accuracy, completeness, format validity, and review effort. A fast, simple selection strategy is valuable only when the resulting outputs meet the application's requirements.

Check your understanding

In the worked example, why does greedy decoding miss Bx?

Answer: It discards B at the first step because A has the higher immediate probability, so it never benefits from the strong continuation x after B.

Related terms

Token
Conditional probability
Beam search
Sampling
Logits
Autoregressive generation
constrained decoding
stopping criterion

Learn next

Draw the four paths in the worked example and calculate their probabilities. Then study beam search to see how retaining multiple partial sequences changes the search. Compare either method with sampling while keeping factual quality separate from model probability.

Sources

Primary source: Jurafsky and Martin, Speech and Language Processing, chapter 13, decoding discussion: https://web.stanford.edu/~jurafsky/slp3/13.pdf . The greedy-selection and sequence-search passages were read. Numerical probabilities, business scenarios, and the route analogy are original teaching examples; no product-specific decoding guarantee is asserted.

Previous: Greedy best-first search Next: Greedy embedding