Why is the task called self-supervised?

Its targets come from the original text rather than from separately assigned human labels.

Must all selected tokens become mask symbols?

No. Corruption recipes can include random replacements and unchanged selected tokens.

Is the original 15% selection rate mandatory?

No. It is a well-known recipe choice; other rates and schemes should be assessed for the model and objective being used.

AI Glossary / Generative AI and language / Intermediate

Masked language modeling

Definition

Masked language modeling is a self-supervised training task in which selected tokens in a text are hidden or altered and a model learns to predict their original identities from context. The original text supplies the target answers, so people do not have to label every training example manually. The task is often used to teach a model useful contextual language representations.

What it means

This is a learning objective, not a single architecture or finished application. The model practices recovering missing information while using visible material before and after each target. Through many examples, it can learn patterns of syntax, meaning, and word usage. The resulting model may later be adapted to classify documents, label entities, or answer questions about supplied text. Success on masked-token prediction and success on a business task are related but distinct: a model that recovers common words well may still need substantial task-specific adaptation and evaluation.

How it works

Start with a tokenized text sequence and select prediction positions. Save the original token IDs as targets, then create a modified input. Run the model on that input and compute a loss at the selected positions, commonly cross-entropy. Optimization adjusts the model to give more probability to the original targets. Different schemes can select individual tokens, whole words, or spans, and can change the selected positions across training passes. In the original BERT recipe, 15% of token positions were selected for prediction. Of those selected positions, 80% were replaced by a mask token, 10% by a random token, and 10% were left unchanged. These are specific design choices from that recipe, not a definition that every masked-language-modeling system must follow.

Three practical examples

A research team pretrains an encoder on a large collection of ordinary text before adapting it to several labeled tasks.
A company continues masked-language-model training on appropriately available technical documents to expose an encoder to specialized terminology, then tests whether downstream classification improves.
A developer compares token-level and whole-word masking on a fixed evaluation design to study how the corruption scheme affects learned representations. Each example concerns how training examples are constructed, rather than simply asking a deployed model to fill a blank.

Think of it this way

Think of making revision exercises by covering selected parts of a textbook while keeping an untouched answer copy. The learner uses the visible context, and the answer copy provides feedback. The analogy explains how training targets can come from the data itself. It does not mean the model learns every fact reliably, or that the remaining context always determines a unique answer. Text can support several plausible completions even though the training example contains only one observed original token.

Worked example

Consider 2,000 eligible token positions and apply the original BERT proportions as an illustrative expected allocation. Selecting 15% gives 300 prediction targets. Of those 300, 240 are expected to receive a mask symbol, thirty a random replacement, and thirty remain unchanged. Relative to all 2,000 positions, these are 12%, 1.5%, and 1.5%. The remaining 1,700 positions are not selected targets in this example. The loss is computed for the 300 selected positions, including the unchanged selected positions; it is not restricted to the 240 visible mask symbols. Random selection means actual counts can vary unless the implementation fixes them. A random replacement may also coincide with the original token, depending on how replacements are sampled. This arithmetic prevents the common confusion between selection rate and visible-mask rate.

Illustrative case study

Fictional case: Pinebridge Search wants better representations of maintenance notes. It continues pretraining an encoder on its document collection, keeping a separate collection for evaluation. An engineer initially reports loss averaged over every sequence position, including positions that should have been excluded from the prediction objective. After correcting the target handling, the team reruns the comparison and then tests document matching on reviewed query pairs. The case demonstrates why a correct training setup and a downstream evaluation are both needed; a falling number from an incorrectly defined loss is not sufficient evidence of improvement.

Common misconceptions and mistakes

Do not confuse content masking with the attention mask that controls permitted interactions between positions. Also avoid training a bidirectional model to predict every visible token from an unchanged copy of the same input and assuming it must learn the intended reconstruction task; it may exploit direct access to the answer. The original BERT mixture deliberately includes some unchanged targets, but this is part of a larger corruption scheme. Another mistake is comparing losses across different tokenizers, masking rates, or evaluation procedures as if they measured exactly the same challenge. Changes in task construction can change the score without a corresponding improvement in downstream utility.

Business use

For a business project, define why additional masked pretraining is needed and what measurable task should benefit. Review the training corpus for relevance, duplication, and separation from evaluation material. Record the tokenizer, masking scheme, selected-target handling, and loss reduction method. Compare the adapted model with the starting model on the actual application, including unfamiliar terminology and document formats. Account for training cost and potential degradation on general text. Domain familiarity is useful only if it improves the intended workflow under realistic testing.

Check your understanding

A training batch contains 100 selected prediction positions, but only eighty contain a visible mask symbol. Should the other twenty automatically be excluded from the loss?

Answer: No. If the scheme selected them as prediction targets, their original-token predictions still contribute according to the defined objective, even when the inputs were randomly replaced or left unchanged.

Related terms

Learn next

Build a small table with original tokens, selected positions, modified inputs, and target labels. Check that non-target positions are excluded as intended. Then distinguish changes in reconstruction loss from changes in a separate downstream task such as message classification.

Reconstruction loss

Sources

Devlin and colleagues, BERT, Section 3.1: https://aclanthology.org/N19-1423.pdf Hugging Face Transformers, Masked language modeling: https://huggingface.co/docs/transformers/tasks/masked_language_modeling

Previous: Masked language model Next: Masking