Is a token always a word?

No. Word tokenizers may separate punctuation, and other tokenizers use subwords, characters or other units.

Does tokenization lowercase the text?

Not inherently. Lowercasing is a separate transformation, even if a particular pipeline performs both.

Can two tokenizers both be reasonable?

Yes, when they follow different documented conventions suited to different tasks.

AI Glossary / Generative AI and language / Advanced

Word tokenization

Definition

Word tokenization divides text into word-level units and, depending on the rules, separate punctuation or other tokens. It establishes the pieces that later language-processing steps will count, label or analyze. A token is an operational unit chosen by a tokenizer, so its boundaries do not always match a reader's intuitive idea of a complete word.

What it means

Splitting at spaces is a possible starting point, but it leaves many decisions unresolved. Should punctuation stay attached? Should a contraction remain one unit? Should an abbreviation or decimal number be split at its periods?
Different applications can require different answers. A search index, a linguistic annotation project and a document word counter may legitimately use different conventions. The important requirement is that the convention fits the task and is applied consistently.
Word tokenization also differs from subword tokenization, where a model can divide one ordinary word into smaller pieces. A count from a word tokenizer therefore should not automatically be used as a language model's token count.

How it works

A tokenizer identifies boundaries using rules, exceptions, learned segmentation or a combination of methods. It may separate punctuation while preserving selected patterns such as decimal numbers. Language and domain conventions influence those choices.
The output can include token text and character spans linking each token to the original input. Retaining alignment helps a later system highlight an extracted phrase or compare annotations without guessing where the token occurred.
Tokenization is distinct from lowercasing, stemming, lemmatization and removing stop words. Those operations change or filter the representation in other ways. Some processing pipelines combine them, but separating their purposes makes it easier to diagnose unexpected results.
A pretrained pipeline should receive tokenization compatible with its training setup. Changing boundaries can change the units that downstream components expect.

Three practical examples

A feedback-analysis system counts product mentions. The team checks whether punctuation attached to a name prevents a match and whether abbreviations need special handling.
An annotation tool highlights words associated with user-selected labels. Character spans allow those labels to remain connected to the exact source text.
An educational word counter documents how it treats contractions and hyphenated expressions. This makes differences from another counter explainable instead of presenting one unexplained number as universally correct.

Think of it this way

Imagine cutting a printed sentence into pieces before sorting them. The cutting rules determine what can be counted or assigned a label afterward.
Cutting carefully does not mean discarding pieces. Punctuation can remain available as its own token, and a record of the original positions can preserve the connection to the sentence. The choice of boundaries is the central issue; later decisions about which pieces to keep are separate.

Worked example

Take the original sentence “Mina paid 12.50 for tea.” Define a teaching tokenizer that separates the final full stop, preserves the decimal number and uses spaces as boundaries elsewhere.
It returns six tokens: “Mina”, “paid”, “12.50”, “for”, “tea” and “.”. A whitespace-only split instead returns five units because “tea.” stays together. A rule that separates every punctuation character would also split the decimal point, producing eight tokens: “Mina”, “paid”, “12”, “.”, “50”, “for”, “tea” and “.”.
None of these counts alone tells us which tokenizer serves the intended task. A price extractor may benefit from preserving the decimal, while a later reconstruction step needs enough information to restore the original spacing and punctuation.
Using zero-based character positions and end-exclusive spans, “12.50” occupies [10, 15), and “tea” occupies [20, 23). The final period occupies [23, 24). These spans refer to the original sentence without the surrounding quotation marks.
The example demonstrates explicit rules rather than claiming that every named library returns this exact output. In particular, a tokenizer's language settings, exceptions and version can affect its decisions.

Illustrative case study

Fictional Rowan Lessons compares vocabulary use across student writing samples. A software update changes how several contractions and hyphenated expressions are split. The dashboard reports a rise in the average token count, which initially looks like a change in writing behavior.
The team reruns a fixed set of texts through both configurations and traces the difference to token boundaries. It keeps the original documents, records the tokenizer version and rebuilds the comparison using one consistent configuration.
The finding does not mean the newer tokenizer is worse. It means the measurement changed. The team adds a short explanation of its counting convention so readers understand what the displayed quantity represents and can compare periods fairly.

Common misconceptions and mistakes

Whitespace is not a universal definition of a word boundary. Writing systems and language conventions differ, and even space-separated text contains exceptions.
Tokenization does not necessarily remove punctuation. Many tokenizers retain punctuation as separate units.
A token list does not automatically preserve all original formatting. Reconstruction requires appropriate spacing or offset information; some tools explicitly preserve it, while simpler outputs may not.
Word tokenization is not semantic understanding. Identifying boundaries does not establish the meaning of an ambiguous term or resolve which person a pronoun refers to.

Business use

Treat tokenization as part of the measurement or model specification. Record its language, rules and version when counts or extracted phrases feed business reports.
Review examples that matter in the actual material: product codes, names, abbreviations, decimal values and mixed-language text. A convention that performs well on ordinary prose may need adjustment for a specialist collection.
Before comparing results across systems, check whether their units match. A difference in vocabulary counts or extraction performance may begin with different token boundaries, rather than with a more sophisticated model later in the pipeline.

Check your understanding

How many tokens does the teaching tokenizer return?

Answer: Six, including the final full stop.

Why does the punctuation-splitting version return eight?

Answer: It divides the decimal number into three units instead of one.

What does [10, 15) identify?

Answer: Characters ten through fourteen, forming “12.50” in the original sentence.

Related terms

Token
Normalization
Lemmatization
Vocabulary
Preprocessing
sentence segmentation
subword tokenization
character offset

Learn next

Create a small collection containing decimals, contractions, abbreviations and product codes. Write the desired boundaries for the intended task, compare a tokenizer's output with them and investigate differences before changing the rules.

Sources

NLTK, tokenizer package documentation: word and punctuation tokenizers, decimal handling and token spans. https://www.nltk.org/api/nltk.tokenize.html spaCy, Linguistic Features, tokenization: language-specific rules, preserved alignment and compatibility between training and runtime tokenization. https://spacy.io/usage/linguistic-features#tokenization The Mina sentence, character spans and fictional reporting case are original teaching illustrations.

Previous: Word shape Next: word2vec-CBOW