Does “bidirectional” mean the model reads a document twice?

Not necessarily. It means representations can use context from both earlier and later visible tokens.

Must every input contain a mask?

The fill-mask interface requires a designated target, but downstream classification or token-labeling applications can use complete text.

Is BERT the only masked language model?

No. It is an influential example, and other models use related objectives with different training choices.

AI Glossary / Generative AI and language / Intermediate

Masked language model

Definition

A masked language model is a language model trained to predict selected hidden or corrupted tokens using the surrounding text. It can use visible context on both sides of a missing token. BERT is a well-known example. The model learns contextual representations that can support language-understanding tasks, as well as producing candidate replacements for masked positions.

What it means

Consider “The parcel was [MASK] yesterday.” The words before and after the blank help identify plausible replacements such as “delivered” or “dispatched.” A masked language model scores vocabulary tokens for the missing position. It does not know which real-world event occurred unless adequate evidence is provided, and even then its prediction is not verification. The term describes the trained model. Masked language modeling describes the training task or objective used to build it. Keeping these separate helps explain why a model trained through fill-in-the-blank exercises can later be adapted to tasks that contain no blanks.

How it works

During training, selected token positions become prediction targets. The input is modified so that the model must use context to recover original tokens. A bidirectional Transformer encoder processes the visible sequence and creates a context-sensitive vector for each position. A prediction head turns the vector at a target position into scores over the vocabulary. Training adjusts parameters to increase the probability assigned to the original target token. Once trained, the same representation network can be combined with a task-specific head for classification, token labeling, or other tasks. An application must load the appropriate head and tokenizer; a classification model and a masked-token prediction model may share a backbone while producing different outputs.

Three practical examples

A writing aid suggests plausible replacements for a missing token in a sentence, leaving the author to decide which matches the intended meaning.
A company adapts a pretrained encoder to classify customer messages into topics using labeled examples.
A document-processing team fine-tunes token-level representations to identify product names or organization names. The second and third applications use representations learned through masked prediction; they do not necessarily insert mask symbols into every live request.

Think of it this way

Think of someone practicing with a damaged newspaper, using the surrounding sentences to infer obscured words. That practice can improve familiarity with patterns useful in later reading tasks. The analogy stops short of human understanding: the model learns numerical relationships through optimization and can favor a common completion over the actual missing word. It also predicts tokenizer units, which may be fragments of words rather than the neat word-sized blanks of a classroom exercise.

Worked example

Imagine a toy tokenizer where “The”, “parcel”, “was”, “delivered”, and “yesterday” are separate tokens. Mask “delivered” and suppose the model assigns 0.60 probability to delivered, 0.25 to dispatched, and 0.15 to all other candidates combined. A top-one fill selects delivered. If the source sentence actually used dispatched, the top-one prediction is wrong even though delivered is grammatically plausible. The loss for the true dispatched token is minus the natural logarithm of 0.25, approximately 1.386. If training later raises its probability to 0.50 on the same example, that loss falls to approximately 0.693. This measures better recovery of the observed token, not proof about a parcel's status. A real tokenizer might split one of these words into several pieces, changing the prediction problem.

Illustrative case study

Fictional case: Elm Archive wants to organize technical support notes. It begins with an encoder pretrained using masked language modeling, then trains a small classification head on reviewed topic labels. The team compares performance on recent notes that were excluded from training and checks unfamiliar abbreviations separately. It discovers that strong masked-word suggestions do not automatically mean good topic classification, so it evaluates the actual categorization task. The case illustrates transfer from pretraining to a business application without treating the pretraining exercise as a complete measure of downstream usefulness.

Common misconceptions and mistakes

A masked language model is not automatically an open-ended conversational assistant. Its basic prediction interface fills selected positions; producing long coherent text requires an appropriate generation procedure or a different model setup. Another mistake is equating a mask token with an attention mask. A mask token marks altered input content, whereas an attention mask controls which positions may interact or be attended to. Predictions at several masked positions also should not casually be treated as a fully validated joint probability for the completed sentence. Plausible individual replacements may form a poor combination when considered together.

Business use

Consider masked-language-model encoders when the application requires analyzing existing text rather than primarily writing long responses. Evaluate the complete downstream system for the relevant languages and document styles. Check sequence-length limits, tokenizer behavior, label definitions, and how long documents are divided. If adapting the model to a specialized domain, compare against the original model and a simpler baseline. Preserve the distinction between a likely linguistic completion and a verified business fact. A missing account status should come from an authoritative record, even if a model can guess a common status label.

Check your understanding

A fill-mask model assigns high probability to “approved” in a contract sentence. Does that establish that the contract was approved?

Answer: No. It indicates a likely token under the model and context. Approval is a real-world status that must be checked against appropriate evidence.

Related terms

Masked language modeling
Contextual embeddings
Token
Fine-tuning
BERT
bidirectional encoder
contextual embedding
prediction head
causal language model

Learn next

Try a sentence with two plausible completions and change only the right-hand context. Observe how that context should affect the prediction. Then compare a fill-mask output with a classification output to understand how a shared representation model can support different task heads.

Sources

Devlin and colleagues, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: https://aclanthology.org/N19-1423/ Hugging Face Transformers, Masked language modeling: https://huggingface.co/docs/transformers/tasks/masked_language_modeling

Previous: Masked autoregressive flow Next: Masked language modeling