Can a span contain one token?

Yes. A selection can be a single token or a longer passage, depending on the task.

Does every span have a label?

No. Boundaries can identify a passage before any category or interpretation is attached.

Some annotation schemes allow overlapping or nested spans, while particular tasks and tools may restrict them.

AI Glossary / Natural language and speech / Intermediate

Span

Definition

In natural language processing, a span is a selected stretch of text identified by its boundaries. It commonly consists of consecutive characters or tokens and may represent a name, phrase, answer, or other relevant passage. The span specifies which text is selected; a separate label can describe what that text represents.

What it means

A document-processing system often needs to point to an exact location rather than return an isolated string. If a report mentions “River House” twice, those two occurrences contain the same words but occupy different spans.
Boundaries make such references precise. A span can be stored with a starting position and an ending position in a particular document representation.
The meaning of those positions must be clear. Token offsets count tokens, while character offsets count units in the text representation used by the software. A pair of numbers without its document and indexing convention is not enough to reconstruct the selected passage reliably.

How it works

Choose the text representation and unit of indexing. Token-based spans depend on how the document was tokenized; character-based spans depend on the exact text and character-counting convention.
Record the start and end boundaries using a documented convention. For example, spaCy uses a start token included in the span and an end token immediately after it. Other formats must be checked rather than assumed identical.
A model or annotation process may then attach a category, role, identifier, or confidence score to the selected text. These attributes describe the span without changing its boundaries.
When the document is edited or normalized, preserve a mapping or recalculate offsets. Otherwise, a previously correct selection may point to different material even though its numerical positions have not changed.

Three practical examples

A named entity recognizer marks the exact words belonging to an organization name before assigning an organization label.
An extractive question-answering system returns the passage containing an answer and its position in the source document.
A document review interface highlights the sentence fragment that supports a proposed record value so a reviewer can inspect its context.

Think of it this way

Think of highlighting a passage in a book and recording its location. The highlight tells you which words were selected, while an attached note says why they matter. The analogy is limited because software uses exact offsets that can become invalid when the underlying text changes, even slightly.

Worked example

Consider the text “Maya visited River House yesterday.” For this exercise, define the tokens explicitly as 0:Maya, 1:visited, 2:River, 3:House, 4:yesterday, and 5:the final period.
Under an inclusive-start, exclusive-end convention, the token span [2, 4) selects tokens 2 and 3: “River House.” Its length is 4 minus 2, or two tokens. The end value 4 does not mean that “yesterday” is included.
The span [2, 5) instead selects “River House yesterday,” which is three tokens. A model could assign an organization label to either selection, but the second boundary would still be wrong for the intended two-word name.
If a preprocessing step inserts “Dr.” before Maya and treats it as one additional token, the original name shifts to [3, 5). Reusing [2, 4) would now select the wrong words.
This exercise shows why boundary correctness, category correctness, and document-version consistency are separate checks.

Illustrative case study

In a fictional project, Cedar Records builds an extraction tool for venue names in event notes. Reviewers find that the system often recognizes the right kind of entity but includes a neighboring date in its highlight.
The team records boundary errors separately from category errors. It also preserves the original text alongside a normalized search version so that a highlight can be traced back to the document people actually read.
The imagined process makes errors easier to diagnose. It does not assume every useful text selection is a named entity, or that a correct-looking copied string proves the stored offsets identify the correct occurrence.

Common misconceptions and mistakes

A span is not necessarily a complete sentence or a syntactic constituent. The selection depends on the task.
Matching text strings do not imply identical spans: the same wording can occur at different positions.
Token indices and character indices are not interchangeable. Even the number of tokens can change with a different tokenizer.
Finally, a single contiguous span cannot directly represent separated pieces while excluding the text between them. Tasks that annotate discontinuous material need multiple spans or another explicitly defined representation.

Business use

Precise spans support inspectable extraction, annotation, search highlighting, and evidence displays. They let people connect a structured prediction with the source material that motivated it.
For quality assessment, define whether a match requires exact boundaries or allows partial overlap. Those measurements answer different questions and should not be reported as though they were equivalent.
Include repeated names, punctuation, text revisions, and differing tokenization in integration tests. These reveal cases where the predicted phrase is plausible but the location displayed to the user is wrong. A useful result needs both a meaningful selection and a reliable connection to the source.

Check your understanding

With exclusive-end token indexing, how many tokens does [4, 7) contain?

Answer: Three: tokens 4, 5, and 6. Token 7 is outside the span. This answer depends on the stated indexing convention.

Related terms

Token
Tokenization
Named entity recognition
extractive question answering
annotation
character offset
coreference
constituent

Learn next

Compare token boundaries with character boundaries in a short document. Then examine exact-match and overlap-based evaluation to understand how a correct label can still accompany an incorrect selection.

Sources

Sources: spaCy official Span documentation, boundary conventions and token versus character attributes: https://spacy.io/api/span . Jurafsky and Martin, Speech and Language Processing, mentions and span-based coreference: https://web.stanford.edu/~jurafsky/slp3/24.pdf . Independent review checked spaCy’s start/end token definitions and separate character-span interface.

Previous: SOV language Next: Span-corruption language-model objective