Is a lemma always the root of a word?

No. It is a chosen canonical form and can contain derivational morphology.

Does lemmatization remove tokens?

Not inherently. It assigns normalized forms; filtering or deduplication are separate operations.

Can one spelling have different lemmas?

Yes. Context and part of speech can distinguish saw as a verb from saw as a noun.

AI Glossary / Natural language and speech / Beginner

Lemma

Definition

A lemma is the canonical or dictionary-style form used to represent a set of related word forms. For example, inspect can serve as the lemma for inspects, inspected, and inspecting when they are verb forms. The process of assigning lemmas is lemmatization. A lemma is not necessarily a root, a unique word meaning, or the shortest string obtained by removing letters.

What it means

Text contains many forms of the same lexical item. Grouping those forms under a lemma can make it easier to search documents or count vocabulary without treating every grammatical variation as unrelated.

The original form still matters. Inspected and inspects can differ in grammatical information even when both map to inspect. A useful representation can therefore retain the surface word, lemma, part of speech, and grammatical features together. Collapsing forms is a purposeful transformation, not proof that the forms are interchangeable in every sentence.

How it works

A lemmatizer uses lexical information, morphological rules, contextual models, or a combination to select a canonical form. Part of speech can be important: saw as a past-tense verb can map to see, while saw as a tool noun normally remains saw. A single spelling can therefore require different outputs in different contexts.

Canonical-form conventions vary across languages and resources. Universal Dependencies treats the lemma separately from grammatical features and notes that language-specific lexicons help determine it. Inflection is commonly normalized, while derivational changes are generally retained: a derived noun need not be reduced to the verb from which it is historically related.

Stemming is a different normalization approach that often removes or changes endings through string rules. It can produce forms that are not ordinary dictionary entries. The distinction is about the intended analysis and method, not a guarantee that a lemmatizer is always correct or that stemming is never useful.

Three practical examples

1. Search: An index connects inspect and inspected through a lemma so a query can find related grammatical forms, while results still display the exact source wording.

2. Corpus analysis: A team counts lemma occurrences to study vocabulary use without splitting common inflectional variants into separate categories.

3. Language learning: A reading tool links an unfamiliar inflected form to a dictionary entry, preserving the sentence context so the learner can understand the particular use.

Think of it this way

Think of several editions of an item filed under a shared catalog heading. The heading helps group them, while each edition retains its own details. A lemma similarly organizes word forms. The analogy is limited because language involves ambiguity and resource-specific conventions; a shared heading does not uniquely identify every sense or grammatical use.

Worked example

Take the original sentence “Mira inspected valves and inspects pumps.” Ignore punctuation, lowercase the words for this exercise, and split on spaces. The six surface tokens are mira, inspected, valves, and, inspects, pumps. All six happen to be distinct surface forms.

With a declared simple English analysis, their lemmas are mira, inspect, valve, and, inspect, pump. There are still six token occurrences, but only five distinct lemma strings because inspected and inspects now share inspect. The normalization changes the distinct-type count from six to five; it does not remove a word occurrence from the sentence.

Compare a separate pair: “Mira saw the mark” and “Mira used a saw.” The first saw is a verb form with lemma see, while the second is a noun with lemma saw. A lookup that maps every saw to see would fail on the tool sense. These are illustrative analyses under stated conventions, not outputs attributed to a particular installed tagger.

Illustrative case study

Fictional case: A service team counts mentions of inspection activity in maintenance notes. Its first report keeps inspect, inspects, and inspected in separate keyword buckets, making the summary cumbersome.

The team adds lemma-based grouping but retains surface forms and grammatical features in the underlying data. It discovers ambiguous words that require context and reviews those mappings before using the report. The final display can group related forms while allowing readers to inspect the original notes. This scenario illustrates a reporting design, not a measured accuracy or productivity gain.

Common misconceptions and mistakes

Do not assume a lemma uniquely identifies meaning. A word such as bank can retain the same lemma string across distinct senses, so lemmatization alone does not solve word-sense disambiguation. Also avoid confusing a lemma with a mathematical theorem called a lemma; this entry uses the linguistic meaning.

Another mistake is stripping derivational structure until only a very short root remains. Under ordinary lemmatization conventions, related lexical items can retain different lemmas. Finally, do not discard original forms when tense, number, emphasis, or exact quotations matter. Normalized search keys and faithful source text serve different purposes.

Business use

Businesses using lemma-based analytics should document the language, resource, capitalization policy, and treatment of ambiguous forms. Evaluate examples from the actual domain, including technical terms and names that general-purpose resources may handle poorly. Compare the usefulness of grouped results with the information lost through normalization. Preserve original text so a reviewer can trace a count or search hit back to its context.

Check your understanding

Why does the worked sentence still have six occurrences after lemmatization?

Answer: Both inspected and inspects remain separate occurrences, even though their normalized strings are both inspect. Only the number of distinct strings decreases.

Related terms

Lemmatization
Root
Part of speech
Word sense
lexeme
surface form
stem
morphological analysis
token and type

Learn next

Select five inflected words from a short text and record their surface forms, lemmas, and grammatical features separately. Add one ambiguous spelling in two contexts. Compare a stemmer with a lemmatizer, checking which distinctions each method preserves or loses for your task.

Sources

Primary resource: Universal Dependencies, Morphology: General Principles, lemma section: https://universaldependencies.org/u/overview/morphology.html . The canonical-form definition, language-specific conventions, separation of features, and retained derivational morphology were read. Sentence analyses, counts, and the reporting case are original examples. Exact outputs depend on the selected language resource and annotation policy.

Previous: Lemke's algorithm Next: Lemmatization