Does NSP write the continuation?

No. It classifies a supplied text pair.

Must the pair contain exactly two grammatical sentences?

No. BERT's training segments can contain multiple sentences.

Do all transformer models use NSP?

No. It is an optional training objective, not a defining property of transformers.

AI Glossary / Probabilistic machine learning / Advanced

Next sentence prediction

Definition

Next sentence prediction (NSP) is a self-supervised classification task that asks whether a supplied second text segment follows a first segment in the source text. It was used alongside masked language modeling in the original BERT pretraining procedure. Despite its name, it classifies a provided pair rather than generating the next sentence.

What it means

The training data supplies its own labels through document order and sampling. A genuine continuation is a positive example; a randomly selected alternative provides a negative example. The model learns to distinguish these constructed cases.
This is not a direct test of whether one sentence logically entails another. A real continuation can introduce a new topic or contradict a preceding claim. Conversely, a sampled negative can sound like a plausible continuation. The target is the label produced by the sampling procedure, not a universal judgment of coherence or truth.

How it works

In original BERT, positive and negative pairs are selected with equal probability. The input includes a classification token, the two text segments and separators. Segment embeddings distinguish the two parts. A classification head uses the classification-token representation to predict the pair's label.
The classification loss is trained alongside the masked-token objective. Gradients update the shared encoder as well as the relevant prediction heads. The so-called sentences can be spans containing more than one ordinary sentence.
Researchers later investigated alternatives. RoBERTa removes NSP while changing the input construction and other training choices. Its findings show that NSP is not a universal requirement for useful language representations; they should not be read as proving that sentence relationships never matter.

Three practical examples

Historical model literacy: a team reviewing a BERT-based system explains why its pretraining included both missing-token prediction and a text-pair classifier. This prevents confusing the classifier with text generation.
Corpus preparation: researchers create positive pairs from adjacent document spans and negative pairs through a documented sampling rule. They check that parsing errors have not turned page headers into apparent continuations.
Objective comparison: an experimental team compares pretraining recipes using held-out downstream tasks. It records differences in text packing and token budgets so a performance change is not attributed to NSP alone without evidence.

Think of it this way

Imagine receiving two strips cut from a document and deciding whether the second originally followed the first. You are choosing between two labels, not writing a missing strip. A convincing match may help, but similar subject matter does not prove that the strips were neighbors in the original document.

Worked example

Consider an original illustrative batch of four text pairs. Their labels are 1, 1, 0 and 0, where 1 means a sampled true continuation. Suppose the model assigns probabilities of IsNext equal to 0.8, 0.6, 0.3 and 0.7. These are stipulated outputs, not measurements from a trained system.
Using a threshold of 0.5 gives predicted labels 1, 1, 0 and 1. Three are correct, so classification accuracy is 3/4, or 75%. There are two true positives, one true negative, one false positive and no false negatives.
Binary cross-entropy for one pair is minus y times log(p), minus (1-y) times log(1-p), using natural logarithms. The four losses are approximately 0.2231, 0.5108, 0.3567 and 1.2040. Their mean is approximately 0.5737 nats per pair. A constant probability of 0.5 would give approximately 0.6931 nats on every pair.
Now change only the fourth probability from 0.7 to 0.95. The predictions and 75% accuracy remain unchanged, but that incorrect pair's loss rises to approximately 2.9957. Mean loss increases to approximately 1.0216. The example shows why accuracy alone misses the cost of confident errors. It does not establish calibration from four examples, and this NSP loss excludes any masked-token loss used in joint pretraining.

Illustrative case study

Fictional company Alder Documents investigates a language encoder for matching support questions with relevant passages. Its researchers study an NSP-style auxiliary objective using the training collection. They reserve entire documents for evaluation, reducing the risk that adjacent passages from one document appear across both partitions.
During inspection, the team finds that randomly selected negatives often concern obviously different products. A classifier could exploit this topic difference without learning the distinctions needed for support retrieval. The team evaluates retrieval quality separately and documents the sampling rule when comparing objectives. This fictional scenario illustrates an evaluation concern; it makes no claim that NSP improved or harmed a deployed service.

Common misconceptions and mistakes

NSP is not next-token prediction, sentence generation or natural language inference. Its output probability relates to the constructed classification problem. A high score is not proof that a passage is true, relevant to a user's question or historically adjacent in every possible document.
Another mistake is interpreting high pretraining accuracy as strong downstream performance. Easy negatives can make the auxiliary task simple. Also, changing the negative sampling strategy changes the learning problem: using carefully selected near-matches is not an identical reproduction of the original recipe. Sentence-order prediction is a related but distinct objective.

Business use

NSP is useful to understand when assessing the training history of text encoders or designing controlled research experiments. A business should judge a resulting model on its intended task, such as retrieval or classification, rather than selecting it because an auxiliary objective sounds appropriate. Document boundaries, duplicated material and representative evaluation data can matter more than the name of the pretraining task.

Check your understanding

Why can a negative pair sound coherent?

Answer: It may be a plausible continuation despite being sampled rather than adjacent in the source.

Why did the example's loss increase without changing accuracy?

Answer: The model became more confident in the same incorrect classification.

What should decide business usefulness?

Answer: Performance on a representative, independently evaluated downstream task.

Related terms

Masked language modeling
Self-supervised learning
Binary cross-entropy
BERT
Sentence order prediction
Natural language inference
Next-token prediction

Learn next

Compare labels for adjacency, entailment and relevance on the same text pairs. Then examine how negative sampling and document-level evaluation splits change what a classifier can learn.

Negative sampling

Sources

Devlin and colleagues, BERT, 2019, Section 3.1 and Appendix A.2: https://aclanthology.org/N19-1423.pdf . Liu and colleagues, RoBERTa, 2019, Section 4.2: https://arxiv.org/abs/1907.11692 . The numerical example and company scenario are original illustrations. BERT Section 3.1/Appendix A.2 and RoBERTa Section 4.2 independently read; losses recalculated.

Previous: Newton-Raphson Next: Next word index