Does a word list need frequencies?

No. Some lists record only membership. A frequency list adds counts or rates.

Are two lists comparable if they have the same number of entries?

Not necessarily. One may contain surface forms while another groups them into lemmas or families, and their source material may differ.

Can a list include multiword expressions?

Some resources do, but that convention should be explicit. An entry such as a product name or fixed phrase is not equivalent to a single whitespace-delimited token.

AI Glossary / Natural language and speech / Beginner

Word list

Definition

A word list is an organized collection of words selected for a stated purpose. In corpus analysis, it often records the distinct word forms found in a collection of texts, sometimes with frequencies or other information about their use.

What it means

The phrase can describe several resources: a frequency list, a spelling vocabulary, a teaching list or a list used by a language-processing tool. Its meaning depends on how entries were selected and what each entry represents.

A list of word forms can count run, runs and running separately. A lemma-based list may group inflected forms under a base entry, subject to analysis of their grammatical use. A word-family list can group additional related forms according to its own criteria.

A word list is not automatically a dictionary. It may provide no definitions, pronunciations or sense distinctions. Nor is a list extracted from one corpus a complete inventory of a language. Its coverage reflects the material and processing choices that produced it.

How it works

To produce a corpus-based list, choose the texts, identify token boundaries and decide what to do with case, punctuation, numbers and repeated material. Count occurrences if frequency is required, then sort or filter the resulting entries.

The pipeline should record whether entries represent surface forms, lemmas or another grouping. It should also distinguish total occurrences from the number of documents containing a word. A repeated template can create a high occurrence count without broad use across independent sources.

NLTK's corpus documentation distinguishes word-list resources from richer lexicons and text collections. A tool can load such resources, but the usefulness of the resulting list still depends on its origin, licensing, language and intended task.

Three practical examples

An editor compiles frequent technical terms from company manuals to identify candidates for a glossary.
A language teacher selects vocabulary from a defined set of course readings and checks which words deserve explanation.
A text-processing team uses a reviewed list of accepted product names to reduce inappropriate spelling corrections.

Think of it this way

Think of an inventory of ingredients found in a set of kitchens. It tells you which items appeared and perhaps how often, but it does not explain every recipe or represent all possible cuisines.

The analogy is limited because the boundaries of a word and its related forms depend on linguistic conventions. The useful lesson is that an inventory is shaped by both the sampled material and the counting rules.

Worked example

Making counting conventions visible

Use the invented text “Tools help teams. Teams share tools.” Lowercase all words and remove sentence punctuation. The token sequence is tools, help, teams, teams, share, tools.

There are six word tokens but four distinct word types. The frequency list is tools: 2, teams: 2, help: 1, share: 1. Alphabetical order would produce a different arrangement without changing the counts.

If a suitable noun lemmatizer groups tools under tool and teams under team, the entry labels change but the total token count remains six. This particular example still has four distinct lemmas.

Now suppose the two sentences are separate documents. Tools occurs twice across the collection and appears in two documents. If both occurrences had been in the first document instead, its token frequency would still be two but its document frequency would be one. The two measures answer different questions.

Illustrative case study

A fictional technical publisher creates a word list from installation guides to plan beginner explanations. The first list is dominated by repeated legal notices and navigation labels.

The team separates recurring boilerplate from instructional text and records that decision. It then reviews frequent words in their original sentences, because a familiar-looking word such as “port” may have a specialized meaning. Rare component names are retained when they are important to the reader's task.

The publisher treats frequency as one signal for editorial priorities, alongside difficulty and importance. It does not claim that the most frequent items automatically form the best teaching sequence. This case illustrates how a list becomes useful through context and review.

Common misconceptions and mistakes

Confusing tokens with types. Repeated occurrences increase token counts without adding distinct entries.
Calling every listed string a word without inspecting tokenization artifacts.
Treating a corpus-specific list as a complete language vocabulary.
Assuming high frequency means high importance or difficulty.
Combining lists that use different case, lemma or word-family conventions without reconciling them.
Equating a stop-word list with words that never matter. A word's value depends on the task.

Business use

Word lists can support terminology work, content analysis and language-learning design. They are simple enough to inspect, but their apparent simplicity can hide consequential choices.

Document the source collection, time period, language and preprocessing. If comparing departments or periods, normalize counts for different corpus sizes and examine whether document types changed. A longer collection naturally offers more opportunities for a word to occur.

For AI workflows, review how a list is used. A spelling aid, retrieval expansion list and restricted command vocabulary have different requirements. Removing an item from one resource does not remove its meaning from language, and adding it does not teach a model every sense. Keep representative contexts so reviewers can understand why entries were included.

Check your understanding

A text contains the word “sensor” twenty times and “cable” once. How many tokens and types do these two forms contribute?

Answer: They contribute twenty-one tokens and two types, assuming those exact forms and no additional normalization distinctions. Frequency does not create new word types.

Related terms

Corpus
Token
Lemma
Frequency distribution
Lexicon
type
word family
document frequency
stop-word list

Learn next

Read tokenization and lemmatization. Build a tiny list from two short documents and report both token and document frequencies. Explain how lowercasing and grouping inflected forms change the entries and their interpretation.

Token

Sources

NLTK project, Corpus Documentation: https://www.nltk.org/howto/corpus.html. Bird, Klein and Loper, Accessing Text Corpora and Lexical Resources: https://www.nltk.org/book/ch02. These primary project and author resources describe corpus access and word-list resources. The tools-and-teams counts and publisher scenario are original illustrations.

Previous: Word error rate Next: Word sense