Do decoder models generate whole words?

They generate tokens, which may correspond to whole words or smaller units.

Can an encoder-decoder model generate autoregressively?

Yes. Its decoder may generate tokens sequentially while using the encoder's representations.

Does greedy decoding find the most probable full sequence?

Not generally, as the worked example demonstrates.

AI Glossary / Statistics and probability / Intermediate

Decoder models

Definition

Decoder models, in the language-modeling sense used here, generate sequences by predicting the next token from preceding context. They are commonly trained with a causal language-modeling objective, which prevents a prediction from using future target tokens. The term also has a broader architectural meaning: a decoder can be the output-generating component of an encoder-decoder system. The intended meaning should therefore be stated.

What it means

A decoder language model assigns probabilities to possible continuations. It can then generate text by selecting a token, appending it to the context, and repeating the process. Tokens may be words, word pieces, punctuation, or other units defined by the tokenizer.

The architecture and the token-selection procedure are different. A trained model produces scores or probabilities; a decoding strategy decides how to turn those scores into an output sequence. Greedy selection, sampling, and beam search can produce different outputs from the same model.

A decoder component in a translation system may additionally use representations supplied by an encoder. A decoder-only model instead handles its text prompt and continuation within its own sequence-processing stack. Neither usage implies that every model with a decoder has identical capabilities.

How it works

During causal language-model training, the text supplies both the context and the next-token targets. Predictions are compared with the actual following tokens, typically using a cross-entropy objective. In a standard transformer decoder, a causal attention mask blocks access to future positions.

Because the training sequence is already known, many position-wise computations can occur together while the mask preserves the prediction constraint. Ordinary autoregressive generation is different: the next generated token becomes part of the context for the following step.

The model factorizes a sequence probability into a product of conditional token probabilities. Additional training can adapt its behavior to instructions or specific tasks, but next-token prediction alone does not establish factual accuracy, instruction following, or reliable reasoning. These qualities need direct evaluation in the intended setting.

Three practical examples

A drafting assistant extends a supplied outline into a proposed paragraph, which the author checks for accuracy and style.
A code-completion system predicts a continuation from earlier code, with testing needed to establish whether the generated program works.
A translation system uses a decoder to generate target-language tokens while attending to encoded source-language information.

Think of it this way

Think of a writer choosing each next piece of a sentence from the text already on the page. Earlier choices change which continuations fit. The analogy explains sequential dependence, but the model is calculating learned patterns rather than independently checking every claim against the world.

Worked example

Consider an original miniature generation example with a fixed prompt and two permitted continuation tokens at the first step. The model assigns A probability 0.6 and B probability 0.4. After A, the probabilities of ending tokens X and Y are 0.51 and 0.49. After B, they are 0.1 and 0.9. Assume every sequence stops after the second token.

Greedy decoding first chooses A, then X, producing AX with probability 0.6 × 0.51 = 0.306. However, BY has probability 0.4 × 0.9 = 0.36, which is higher. The other possibilities are AY with probability 0.294 and BX with probability 0.04. All four probabilities sum to one.

This example shows that choosing the most probable token at each step need not yield the most probable complete sequence. It does not establish that BY is the best answer for a user: sequence probability is a property of the stipulated model, not a measure of truth or usefulness.

If sampling directly from these probabilities, AX is one possible outcome rather than a guaranteed result. Changing the generation procedure can change the output without changing any learned model parameters.

Illustrative case study

In this fictional case, Clearline Support tests a decoder language model for drafting replies to routine service questions. Staff provide a verified policy excerpt and the customer's question as context. The generated reply remains a draft until reviewed.

The team evaluates whether replies preserve conditions and exceptions, not merely whether they sound fluent. It also checks how generation settings affect omitted details and unsupported additions. Errors lead to changes in source preparation, prompts, training choices, or workflow controls as appropriate.

The invented case illustrates evaluation: a decoder architecture does not guarantee dependable customer communication.

Common misconceptions and mistakes

The word causal in causal language modeling refers to the direction of information available for prediction. It does not mean that the model has identified causal relationships in the scientific sense.

A decoder model is not the same thing as a decoding algorithm. A different sampling temperature or search method changes token selection rather than turning one architecture into another.

Predicting tokens accurately on a benchmark is useful evidence about that benchmark, but does not prove every generated statement. Models can produce unsupported details, repeat biases, or mishandle unfamiliar inputs. Also, the broader word decoder includes systems outside autoregressive text generation, such as components that reconstruct data from latent representations. This entry uses the stated language-modeling scope.

Business use

Businesses can use decoder models for drafting, summarization, extraction expressed as text, and other sequence-generation tasks when evaluated appropriately. Selection should consider actual task performance, source handling, latency, cost, and the consequence of incorrect output.

Test the complete workflow, including tokenizer, prompt format, generation settings, context limits, and output validation. Record model and configuration versions so comparisons remain meaningful. Where output must follow a schema, validate structure separately from factual correctness. A well-formed response can still contain an unsupported value, just as a fluent sentence can still misstate a policy.

Check your understanding

Check your understanding: What prevents a training prediction from seeing future targets in a standard causal transformer?

Answer: Its mask and target alignment.

Why can generation depend on an earlier generated mistake?

Answer: That token becomes context.

What distinguishes the model from decoding?

Answer: One supplies probabilities; the other selects outputs.

Does high sequence probability establish truth?

Answer: No.

Related terms

Learn next

Examine a causal attention mask and shifted training targets. Compare several decoding strategies using a small probability tree before evaluating them on a real task.

Attention mask

Sources

Sources: Hugging Face Transformers glossary, Decoder models and causal language modeling, https://huggingface.co/docs/transformers/glossary. Vaswani and colleagues, Attention Is All You Need, decoder architecture and masking, https://arxiv.org/html/1706.03762v7. The probability tree and fictional support case are original. The entry explicitly distinguishes decoder language models from the broader decoder-component meaning.

Previous: Decoder Next: Decoder-only model