Does every model use a token named [SEP]?

No. Models use different conventions, and some keep source and target in separate inputs.

Is a separator the same as an end-of-sequence marker?

Their functions differ; any reuse is a model-specific design choice.

Does adding a separator train the model?

No. It supplies structure within a broader training or inference format.

AI Glossary / Natural language and speech / Advanced

Sentence separation

Definition

In the supplied sequence-generation context, sentence separation means marking the boundary between a source sequence and its target sequence, commonly with a special separator token. The marker helps represent where the conditioning text ends and the output sequence begins. This meaning is distinct from sentence segmentation, which finds sentence boundaries inside ordinary text.

What it means

A model trained on paired examples needs a consistent representation of which text is the input and which text it should produce. A translation pair, for example, contains a source and a target even when they have different lengths and word orders.
The source discussion introduces a separator when concatenating these sequences for a simplified recurrent encoder - decoder explanation. The idea concerns sequence roles, not punctuation alone. A period in the source can be normal content, while a designated separator has a structural role in the model's input format. The exact symbol and token identifier depend on the model and tokenizer.

How it works

A preprocessing pipeline encodes the source, inserts or otherwise represents the required boundary, and prepares the target according to the model's training convention. During autoregressive generation, each target prediction depends on the source representation and the preceding target context.
The separator is different in purpose from an end-of-output marker, padding, and a beginning-of-sequence marker, even when a particular system reuses tokens for some purposes. Follow that system's actual format rather than assuming one spelling has the same meaning everywhere.
For a concatenated causal training example, labels must be shifted so each position predicts the next token. A target-only objective can mask source predictions. Other architectures use separate encoder and decoder inputs, so literal concatenation is not a universal requirement for sequence-to-sequence learning.

Three practical examples

A translation dataset distinguishes the source sentence from its reference translation. This helps the training pipeline assign the intended prediction targets rather than learning an accidentally misaligned pair.
A question-answering experiment marks the transition from a supplied question to its answer in a model-specific format. A boundary marker alone does not determine whether the answer is correct.
A summarization pipeline preserves the distinction between a document and its reference summary. The source can contain many sentences, demonstrating that the structural boundary is not necessarily the end of just one grammatical sentence.

Think of it this way

Think of a worksheet with a prompt area and a response area. The dividing line tells you which area you are reading; it does not answer the prompt. A separator token plays a comparable formatting role, but the model must have been trained to use that format. Drawing a new line in an unfamiliar place does not automatically teach a new task.

Worked example

Use an invented word-level vocabulary in which [SEP] and [EOS] are each one dedicated token. The source is item available and the target wording is in stock. The complete training sequence is: item, available, [SEP], in, stock, [EOS]. It has six tokens: two source tokens, one separator, two target-word tokens, and one end marker.
For next-token training, the five inputs are item, available, [SEP], in, stock. Their aligned labels are available, [SEP], in, stock, [EOS]. In this toy target-only objective, mask the first two labels. The three scored predictions are in after the source and separator, stock after in, and [EOS] after stock.
Suppose the model assigns those correct next tokens conditional probabilities 0.6, 0.5, and 0.8. The probability of this complete target sequence, including termination, is 0.6 × 0.5 × 0.8 = 0.24. Its negative log-likelihood is approximately 1.4271 natural-log units, or 0.4757 averaged over three scored positions. These invented values illustrate alignment and scoring; they do not measure translation quality or describe a particular tokenizer.

Illustrative case study

Fictional team StockPhrase trains a short product-status rewriting model. Its training examples use a dedicated source - target boundary, but a deployment change replaces the required format with an ordinary punctuation character. The team notices inconsistent output behavior and investigates formatting before changing the model's capacity.
They compare token identifiers, boundary placement, target-label shifts, and termination rules between training and inference. A small inspected example accompanies each preprocessing change, and evaluation uses the same declared format as deployment.
This fictional scenario illustrates a consistency check. It does not claim that every formatting mismatch causes the same failure or that repairing a separator guarantees useful generated text.

Common misconceptions and mistakes

Sentence separation in this context is not the task of splitting prose after periods. It also does not mean source and target must contain the same number of sentences or tokens.
Do not assume that typing the visible characters [SEP] creates the special token expected by every model. The tokenizer and model configuration determine their representation. Similarly, an end marker is not automatically a valid replacement for the source - target separator.
Avoid scoring misaligned labels or allowing future target tokens to leak into a causal prediction. A well-formed separator does not repair an inappropriate attention mask, incorrect pairing, or missing target supervision. Check the complete data path.

Business use

Clear source - target boundaries help teams reproduce generation experiments and diagnose preprocessing errors. Record the tokenizer, input template, truncation policy, and scored target region together. Inspect what happens when the source approaches the length limit: truncation should not silently remove the structural information on which the chosen setup relies. Measure output quality separately from formatting correctness, because both are necessary and neither proves the other.

Check your understanding

How many positions are scored in the toy target-only example?

Answer: Three: in, stock, and [EOS].

Why are the first two next-token labels masked?

Answer: They predict source continuation and the separator rather than the target under this explicitly chosen objective.

Why is the target probability 0.24?

Answer: It is the product of three successive conditional probabilities, including the probability of terminating the output.

Related terms

Teacher forcing
Tokenization
Separator token
encoder–decoder
sequence-to-sequence learning
autoregressive generation
end-of-sequence token
loss masking

Learn next

Inspect one paired example before and after tokenization. Write out the model inputs, shifted labels, attention constraints, and loss mask. Then contrast this arrangement with a model that receives source and target through separate encoder and decoder interfaces.

Tokenization

Sources

Jurafsky and Martin, Speech and Language Processing, Chapter 14, section 14.7, the explicitly indexed sentence-separation marker in the RNN encoder–decoder discussion: https://web.stanford.edu/~jurafsky/slp3/14.pdf Hugging Face Transformers, Tokenizer documentation, separator and other special-token roles: https://huggingface.co/docs/transformers/main_classes/tokenizer The status-rewriting example, probabilities, and StockPhrase scenario are original illustrations.

Previous: Sentence-permutation denoising Next: Sentiment