AI Glossary / Statistics and probability / Advanced
n-gram model
Definition
An n-gram model estimates the probability of the next token using only the preceding n-1 tokens. Multiplying its conditional probabilities gives a probability for a token sequence under the model.
What it means
A bigram model uses one previous token; a trigram model uses two. Tokens may be words or other chosen units. The model captures local sequence patterns, but its fixed context does not preserve an arbitrarily long history.
How it works
Count observed continuations for each context and normalize them. An unsmoothed maximum-likelihood estimate divides a continuation count by the total count of continuations from that context. Smoothing redistributes probability so that unobserved continuations can receive positive probability.
Specify tokenization, vocabulary, and sentence boundaries consistently. Evaluate predictions on held-out data. Perplexity summarizes average predictive surprise, but comparisons require compatible tokenization and scoring conventions.
Three practical examples
- A fictional writing aid uses a small word-based model to rank likely continuations in a narrow collection of routine messages.
- A fictional teaching exercise compares bigram and trigram predictions to show what changes when the context grows.
- A fictional text-processing team uses an n-gram model as a transparent baseline before evaluating a more complex predictor.
Think of it this way
Imagine completing a sentence while seeing only a small window of the words just written. A wider window can distinguish more contexts, but each exact context may appear less often in the examples available for learning.
Worked example
Use an original training collection containing four sentences: "we ship today" twice, "we ship tomorrow" once, and "they ship today" once. Lowercase all words and use spaces to separate tokens.
Add a start marker before each sentence and an end marker after it. The start marker supplies context but is not itself a predicted output in this example. The end marker is predicted and counts as a scored token.
For an unsmoothed bigram model, the first word is "we" in three of four sentences and "they" in one. Therefore P(we given start)=3/4 and P(they given start)=1/4.
Every occurrence of "we" or "they" is followed by "ship", so both corresponding conditional probabilities are one. Among the four occurrences of "ship", three are followed by "today" and one by "tomorrow". Thus P(today given ship)=3/4 and P(tomorrow given ship)=1/4.
Every "today" and "tomorrow" ends its sentence, so the conditional probability of the end marker after either is one.
The model probability of the complete sentence "we ship today", including its end marker, is (3/4) times one times (3/4) times one =9/16. This is not the sentence's empirical frequency of two out of four. A bigram model combines local transitions rather than assigning each whole sentence its observed relative frequency.
The model assigns "they ship tomorrow" probability (1/4) times one times (1/4) times one =1/16, even though that exact sentence never appeared in training. All of its local transitions were observed, so the model can recombine them.
There are four possible complete sentences under this particular unsmoothed model. Their probabilities are 9/16 for "we ship today", 3/16 for "we ship tomorrow", 3/16 for "they ship today", and 1/16 for "they ship tomorrow". They sum to one.
By contrast, a sequence requiring "we" immediately after "ship" has zero probability under the unsmoothed model because that transition was never observed. This is a limitation of the estimate, not evidence that the sequence is impossible in all language.
For a smoothing illustration, fix the predicted vocabulary to six tokens: "we", "they", "ship", "today", "tomorrow", and the end marker. The start marker is excluded from this output vocabulary.
Add one to every possible continuation count. After "ship", the denominator becomes four observed continuations plus six added counts, or ten. The probability of "today" becomes four tenths, "tomorrow" becomes two tenths, and each of the other four output tokens receives one tenth. These probabilities sum to one.
The smoothed probability of "we" after the start marker is four tenths. The probability of "ship" after "we" is four ninths, because there were three observed continuations from "we". The probability of ending after "today" is also four ninths.
Consequently, the smoothed probability of "we ship today" is (4/10)(4/9)(4/10)(4/9)=64/2025, approximately 0.03160. The large decrease from 9/16 shows how much this illustrative smoothing rule spreads mass across previously unseen transitions. It is a teaching calculation, not a claim that add-one smoothing is the best choice.
For a separate held-out occurrence of "they ship tomorrow", the unsmoothed probability is 1/16 across four scored tokens, including the end marker. Its perplexity is (1/16) raised to the power -1/4, which equals two. Counting only three tokens would produce a different number, so the convention must be stated.
Finally, consider a trigram prediction after the two-word context "we ship". That context occurs three times, followed by "today" twice and "tomorrow" once. Its unsmoothed probability of "today" is 2/3, whereas the bigram probability after "ship" alone is 3/4. More context changes which observations contribute to the estimate.
Illustrative case study
Fictional company Cedar Messages builds this small model as a training demonstration. Reviewers inspect the counts and compare the unseen complete sentence with the unseen local transition.
For a real baseline experiment, the team separates training and evaluation material and records the vocabulary and boundary rules. It evaluates whether suggestions help the intended task rather than treating a probability score as proof of usefulness. This is a fictional scenario.
Common misconceptions and mistakes
An n-gram model uses n-1 preceding tokens to predict the next one.
An unseen complete sentence can have positive probability.
Zero estimated probability need not mean impossibility.
Smoothing changes probabilities of observed continuations too.
A larger n is not automatically better with limited data.
Business use
A transparent baseline can expose data and evaluation problems early. In the fictional message project, inspect whether the training language resembles the intended setting and whether useful continuations receive appropriate scores. Document scoring conventions so comparisons remain meaningful.
People also ask
Does a trigram use three previous tokens?
No, two.
Can it generate an unseen sentence?
Yes.
Is an n-gram model the same as a transformer?
No.
Check your understanding
What is P(today given ship) before smoothing?
Answer: Three quarters.
What is the unseen complete sentence's probability?
Answer: One sixteenth.
What is its four-token perplexity?
Answer: Two.
Related terms
Learn next
Compare smoothing and interpolation methods, then evaluate n-gram models on consistently tokenized held-out text.
More in Statistics and probability
- Multivariate random variable - A multivariate random variable, also called a random vector, is an ordered collection of random variables defined on the same probability space.
- Multivariate regression - Multivariate regression models several response variables using one or more predictors.
- Multivariate statistics - Multivariate statistics studies data in which several variables are recorded together for each observational unit.
- N-of-1 trial - An N-of-1 trial compares interventions within one individual across repeated periods.
- Naive Bayes classifier - A naive Bayes classifier predicts a class by combining its prior probability with feature evidence under a simplifying assumption: features are mutually...
- Natarajan dimension - Natarajan dimension measures the capacity of a multiclass hypothesis class.
Sources
Jurafsky and Martin, Speech and Language Processing, Chapter 3, N-gram Language Models: https://web.stanford.edu/~jurafsky/slp3/3.pdf . Training collection, counts, probability calculations, and company scenario are original.