Does stemming always produce a real word?

No. An internal stem such as poni can be valid output for a specified algorithm.

Is lemmatization always better?

No. It offers a different linguistic treatment, and its usefulness depends on the task, resources, and evaluation.

Should every token be stemmed?

Not automatically. Some fields or token types require exact matching or different processing.

AI Glossary / Graphs and symbolic AI / Intermediate

Stemming

Definition

Stemming is a text-processing technique that reduces word forms to shared stems, often by applying rules that remove or replace endings. It is commonly used in search and text analysis so that some related forms can match the same index term. A stem is an operational matching form and does not have to be a dictionary word.

What it means

A searcher may type a singular word while a document contains its plural, or use one grammatical form while the document uses another. Stemming attempts to reduce some of those differences before matching.
It does not establish that two words have the same meaning. It also differs from lemmatization, which aims to identify a lexical base form using linguistic information. A stemmer typically emphasizes useful conflation through explicit rules. The appropriate choice depends on language, the collection, and what users need to retrieve. More aggressive reduction can create both additional useful matches and unwanted collisions.

How it works

A typical pipeline separates text into tokens, applies relevant normalization, and runs a chosen stemmer on the tokens intended for stemming. The resulting forms can be stored in a search index. Query terms need compatible processing if they are meant to match that index.
Rule-based stemmers can apply several ordered stages with conditions, rather than simply cutting off any familiar ending. Martin Porter's English algorithm, for example, uses suffix rules whose applicability depends on the remaining word structure. Different algorithms or implementations can produce different outputs. Record the exact choice, and keep original text for display because stems may be unsuitable as user-facing words.

Three practical examples

A document-search prototype can match selected singular and plural forms through a shared stem.
A text-classification experiment can compare features based on original tokens with features based on stemmed tokens.
A multilingual search project can evaluate language-specific stemming rather than applying English suffix rules to every document.

Think of it this way

Think of placing related word labels into a common filing drawer so one lookup can find several variants. The drawer label may be shorthand rather than a normal word. The analogy is limited because a mechanical filing rule can combine labels that people would prefer to keep separate, or fail to combine forms that belong together.

Worked example

Consider an original miniature search collection with four documents. D1 says “cat care,” D2 says “cats indoors,” D3 says “cat sculpture,” and D4 says “dog care.” Suppose a user searching for practical cat-care information submits the query “cat.” Human relevance judgments mark D1 and D2 as relevant and D3 and D4 as irrelevant.
Use lowercase token matching and ignore ranking for this demonstration. Without stemming, the exact token cat retrieves D1 and D3. There are two retrieved documents, one relevant. Precision is 1/2 = 50%, and recall is 1/2 = 50% because only one of the two relevant documents was found.
Now apply the Porter treatment that maps cats to cat, while cat remains cat. The query retrieves D1, D2, and D3. Two of the three retrieved documents are relevant, so precision is 2/3, about 66.7%. Recall is 2/2 = 100%.
This tiny example shows a useful additional match. It does not prove that stemming always improves precision or recall on a real collection. D3 remains an unwanted result because stemming has not understood the user's care-related intent. A richer query, ranking model, or other retrieval design may be needed to address that issue.
Also consider the documented Porter step mapping ponies to poni. The output illustrates why a stem need not be an ordinary English word. It is useful as an internal matching form if compatible processing brings the intended variants together; it should not automatically replace the original wording displayed to readers.
The evaluation must separate algorithm behavior from relevance judgments. The arithmetic follows from the explicitly supplied labels and retrieval sets. Different judgments about whether a sculpture document is relevant would change the reported precision without changing the stemmer at all.

Illustrative case study

In a fictional knowledge-base project, an editor notices that searches miss articles containing plural variants. The team tests stemming on a labeled set of representative queries and compares retrieved results with its existing index.
It separately inspects product names and technical identifiers, where altering a token could damage an exact lookup. The team retains original text for snippets and documents its processing settings. This fictional case describes a controlled evaluation; it does not claim a measured improvement for any real website.

Common misconceptions and mistakes

Stemming is not synonym expansion, spell correction, or full language understanding. Two differently spelled words with related meanings may remain separate, while unrelated uses can still share a stem.
Do not assume that every stem is a linguistic root, or that removing more characters makes a stemmer better. Overstemming merges distinctions that matter; understemming leaves desired variants separate. Avoid comparing two systems as if stemming were the only difference when tokenization, normalization, ranking, or relevance labels also changed.

Business use

The practical question is whether stemming helps users find the intended content in the actual collection. It can reduce avoidable mismatch, but it should be evaluated against the search tasks that matter.
Maintain a small, reviewed query set with expected relevant documents. Compare retrieval coverage, unwanted matches, and user-visible result quality. Consider exact-match fields for identifiers and names where appropriate. Preserve the algorithm and version in the search configuration so that later reindexing or library changes do not silently alter matching behavior.

Check your understanding

Why does retrieving D3 after stemming not demonstrate a bug in the suffix rule?

Answer: D3 already contains cat. The mismatch concerns the user's intended topic, which stemming alone does not resolve.

Related terms

Lemmatization
Tokenization
Information retrieval
Inverted index
Precision
Recall
Text normalization
Overstemming
Understemming

Learn next

Next, add a fifth document containing cats that your reviewers judge irrelevant. Recompute the stemmed retrieval precision: two relevant results among four retrieved gives 50%, while recall remains 100%. Explain why adding a matching document can change precision without changing recall.

Sources

Primary sources: Martin Porter's official algorithm page: https://tartarus.org/martin/PorterStemmer/ ; Snowball's Porter algorithm specification: https://snowballstem.org/algorithms/porter/stemmer.html ; Manning, Raghavan, and Schütze, Introduction to Information Retrieval, stemming and lemmatization: https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html . The collection and relevance calculations are original.

Previous: Stem and leaf plot Next: Step cost