Can it use an entire document?

Yes, if that document is included before the generated output and fits the supported context arrangement.

Must training generate one token at a time?

No. Causal masking can permit parallel training computations without exposing later targets.

Does causal mean factually reliable?

No. The term describes conditioning, not truth verification.

AI Glossary / Generative AI and language / Advanced

Unidirectional language model

Definition

A unidirectional language model predicts tokens using context from one direction in a sequence. In the usual left-to-right setting, it predicts the next token from preceding tokens without looking at the later tokens being predicted. This is commonly called causal language modeling. The word causal here describes the permitted information flow; it does not mean that the model has established cause-and-effect relationships in the world.

What it means

Consider an unfinished sentence. A left-to-right model assigns probabilities to possible continuations using the prefix available so far. Once a token is selected, that token becomes part of the context for the following prediction.
The restriction concerns position in the model's input sequence. Information written later in an original document can still influence an answer if it has been placed earlier in the prompt, before the answer starts. A model can therefore receive a complete document and then generate a summary without violating left-to-right prediction.
Unidirectional does not mean remembering only one preceding token. The usable history depends on the model and its context limits.

How it works

For tokens x1 through xn, a left-to-right language model represents a sequence probability as a product of conditional probabilities: each token is conditioned on the preceding prefix. A beginning-of-sequence convention handles the first prediction.
During training, observed text supplies the target tokens. In a causal Transformer, an attention mask prevents a position from accessing later input positions. Training can process many positions in parallel while preserving that restriction; parallel computation does not imply access to future target tokens.
During ordinary autoregressive generation, the system chooses a next token, extends the prefix and repeats. The choice may use greedy selection or a sampling strategy. Those choices affect the generated path without changing the fundamental direction of conditioning.

Three practical examples

A writing assistant suggests a continuation after the text a user has typed. Its evaluation checks whether useful suggestions appear with the same prefix information available in actual use.
A document assistant receives a source passage followed by a request for a summary. The summary tokens can use that passage because it precedes them in the supplied sequence.
A teaching exercise trains a small character model to complete short patterns. Students compare permitted prefix information with a deliberately incorrect setup that exposes the target character during prediction.

Think of it this way

Imagine reading a story through a window that reveals the text up to your current position. You can consider everything visible in the window, but you cannot use the hidden continuation to justify your next guess.
If someone puts background notes before the story, those notes are also visible. The analogy helps distinguish a sequence-position restriction from a claim that the system knows nothing beyond the immediately preceding word.

Worked example

Use an original toy vocabulary in which each displayed word is one token. Given the prefix “The parcel,” suppose the model assigns probability 0.6 to “arrived” and 0.4 to “left,” with all other next tokens assigned zero for this simplified exercise.
After “The parcel arrived,” let the probability of “today” be 0.5. After “The parcel left,” let the probability of “today” be 0.9. Conditional on the original prefix, the two-token continuation “arrived today” has probability 0.6 × 0.5 = 0.30, while “left today” has probability 0.4 × 0.9 = 0.36.
Greedy selection starts with “arrived” because 0.6 exceeds 0.4. This shows why choosing the most probable next token does not necessarily select the most probable complete continuation. The example compares only these two continuations and does not establish which continuation is globally best.
The numbers also say nothing about an actual parcel. They are model probabilities in a constructed example, not evidence that a delivery occurred. A model can prefer a fluent continuation that is factually wrong.

Illustrative case study

Fictional Willow Writing evaluates an autocomplete feature. Its first experiment reports excellent prediction results, but a review finds that the test input accidentally includes the word being predicted. The measurement therefore does not reflect the intended prefix-only task.
The team corrects input and target alignment and checks the attention restriction. It separately evaluates suggestion acceptance and factual support where the text contains claims. The corrected results are less impressive but describe the real task more honestly.
For a document-summary feature, the team places the complete source before the requested output. This is a legitimate use of context, unlike leaking the target into its own prediction. The distinction depends on the intended task and information available at deployment, not simply on whether a word appeared later somewhere in the original file.

Common misconceptions and mistakes

Unidirectional does not mean the system can only produce one type of text.
Causal attention is not causal inference. It prevents particular information paths within the sequence; it does not validate explanations of real-world causes.
Tokens are not always whole words. A real tokenizer may split a word into several pieces, so word-based probability exercises are simplifications.
A longer available prefix does not guarantee correct use of every detail. Evaluate whether the model follows the relevant information rather than assuming that inclusion in context ensures success.

Business use

For a generated-text workflow, specify what information precedes the output and how that information is selected. Missing source material and misleading context can matter as much as the generation mechanism.
Measure task outcomes alongside language-model scores. A lower prediction loss alone does not establish that summaries are accurate or suggestions save editing time.
Record the model, prompt construction and decoding settings used for evaluation. Changing any of these can change outputs, making an earlier result a poor description of the revised workflow.

Check your understanding

Why can a summary use a document's final paragraph?

Answer: That paragraph can precede the summary in the model's input.

Which illustrated continuation has the higher two-token probability?

Answer: “Left today,” at 0.36.

Why does greedy selection begin differently?

Answer: It compares next-token probabilities, where “arrived” has the higher value.

Related terms

Token
Attention mask
Decoding
Bidirectional language model
Context window
Causal language model
autoregressive generation
next-token prediction

Learn next

Trace a short sequence position by position, listing the information each prediction may access. Then compare greedy selection with alternative continuation paths and distinguish model likelihood from factual correctness.

Sources

Google, Machine Learning Glossary, unidirectional language-model terminology. https://developers.google.com/machine-learning/glossary Hugging Face Transformers v5.4.0, Causal language modeling. https://huggingface.co/docs/transformers/v5.4.0/en/tasks/language_modeling Google, LLMs: What's a large language model?, attention direction and Transformer context. https://developers.google.com/machine-learning/crash-course/llm/transformers The parcel probabilities and fictional evaluation case are original teaching illustrations.

Previous: Unicode Next: Unification