Are classes always learned?

No. They can also be predefined.

Does every version use one class per word?

The basic model here does; more elaborate models can use different membership assumptions.

Is this the same as a neural embedding?

No. A discrete class assignment differs from a learned continuous vector representation.

AI Glossary / Natural language and speech / Advanced

Class-based n-gram

Definition

A class-based n-gram is a language model that predicts words using groups of words called classes. Instead of estimating every word sequence separately, it models sequences of classes and the probability of each word within its class. This shares statistical information across words assigned to the same group.

What it means

An ordinary word bigram predicts the next word from the preceding word. A class-based bigram first predicts the next word's class from the preceding word's class, then assigns a probability to the particular word within that predicted class.
For example, several drink names might belong to one class. Evidence about one drink can then help estimate patterns involving other members. This can be useful when individual word combinations occur rarely in training data. The benefit comes with a tradeoff: grouping words together removes some distinctions between their individual contexts.
Classes need not be grammatical labels such as noun or verb. They can be supplied by a designer or learned from distributional patterns. A learned class should not automatically be interpreted as a group of interchangeable synonyms.

How it works

In the basic model, every vocabulary word belongs to exactly one class. For an n-gram, the model conditions the next class on the preceding n−1 classes. It multiplies that class-transition probability by the probability of the proposed word within its own class.
For a bigram, this gives P(next word | history) = P(next class | previous class) × P(next word | next class), as a modeling assumption. It does not claim that actual language always has this restricted dependency structure.
The class and word probabilities can be estimated from training counts. For example, a word's within-class probability can be its count divided by the total count of words in that class. Estimating transitions requires corresponding class-sequence counts and an explicit treatment of sentence boundaries. Smoothing can address missing or unreliable counts.
Class discovery is a separate part of the process. Brown clustering is a historical approach that groups words using their distributional relationships. A greedy clustering procedure need not find a globally optimal partition, and a useful partition for one corpus may not transfer unchanged to another.

Three practical examples

A small text prediction system groups related location names so that rare locations can benefit from patterns observed for more frequent members of the same class.
A researcher compares word bigrams with class bigrams on held-out text to investigate whether parameter sharing improves predictions under limited training data.
A speech recognition experiment uses class-based probabilities as one component in scoring candidate word sequences, while separately evaluating the final recognized output.

Think of it this way

Think of a shop estimating demand first for a product category, then for individual products within it. Category information supports estimates for infrequently purchased items. However, treating all products as if they share the same surrounding purchasing patterns can hide important differences.

Worked example

Consider an original toy vocabulary containing four words. Class DRINK contains tea and coffee; class FOOD contains bread and rice. These are stipulated teaching classes, not the output of a trained clustering algorithm.
Suppose the preceding word belongs to DRINK. The class bigram assigns probability 0.3 to DRINK next and 0.7 to FOOD next. Within DRINK, tea has probability 0.6 and coffee 0.4. Within FOOD, bread has probability 0.25 and rice 0.75.
The resulting next-word probabilities are:
Tea: 0.3×0.6=0.18.
Coffee: 0.3×0.4=0.12.
Bread: 0.7×0.25=0.175.
Rice: 0.7×0.75=0.525.
The four probabilities sum to 1. Rice is the most probable individual next word in this particular model, and the combined probability of food words remains 0.175+0.525=0.7.
Because tea and coffee share a class, this simple class bigram produces the same next-word distribution after either word. That is the sharing mechanism and also a limitation. It cannot express a preference for bread specifically after coffee while preferring rice specifically after tea, unless the model is extended or the grouping changes.
A word pair absent from training can still receive positive probability if its class transition and within-class word probability are both positive. But class grouping does not automatically eliminate every zero probability. A zero class-transition probability still makes all words reached through that transition impossible under the unsmoothed model.
This calculation covers one prediction step over a deliberately restricted vocabulary. A complete sentence model also needs its other transition distributions and boundary conventions.

Illustrative case study

Fictional example: PhraseShelf develops a predictor from a small collection of maintenance notes. Some equipment names occur too rarely to estimate their individual surrounding word patterns reliably. The team experiments with grouping names by observed contexts and comparing the resulting model with a word-based baseline.
Reviewers inspect both average predictive performance and specific failures. They find a hypothetical case where grouped names require different following expressions, showing why sharing can lose useful information. The team treats this as a modeling tradeoff to test, not proof that either approach always wins. This scenario reports no measured performance gain.

Common misconceptions and mistakes

A class-based n-gram is not simply a word n-gram with labels added afterward. The classes affect the probability factorization itself. Nor does a class necessarily correspond to a human-readable topic or part of speech.
Sharing can reduce sparsity, but it does not guarantee improved accuracy. Poor groupings can erase distinctions that matter. Another mistake is assuming that increasing n restores all lost word-level information: longer class histories still omit distinctions among words within the same classes in this basic model.

Business use

The concept helps teams understand statistical sharing, compact language models, and the limitations of categorical abstraction. Compare models on appropriately separated evaluation data, inspect domain changes, and measure the downstream task as well as predictive probability. Record the vocabulary, class mapping, smoothing method, and treatment of unknown words so results can be reproduced.

Check your understanding

Why is rice assigned probability 0.525?

Answer: Its class probability 0.7 is multiplied by its within-class probability 0.75.

Why are predictions identical after tea and coffee here?

Answer: Both histories reduce to the same preceding class.

Can an unseen word pair receive positive probability?

Answer: Yes, when the two relevant factors are positive.

Related terms

Learn next

Recalculate the example after changing the within-class food probabilities. Then compare predictions from word and class bigrams, identifying which distinctions are shared and which are lost.

Sources

Brown and colleagues, Class-Based n-gram Models of Natural Language, Computational Linguistics, 1992, class-model formulation and clustering: https://aclanthology.org/J92-4003.pdf

Previous: Class activation mapping Next: Class incremental learning