AI Glossary / Natural language and speech / Intermediate
Add-k
Definition
Add-k is a probability-smoothing method that adds the same positive amount k to every possible outcome's observed count before normalizing. In a count-based language model, it gives a nonzero probability to vocabulary words that were not observed after a particular context. The denominator must increase too, so the probabilities still sum to one.
What it means
A small training sample cannot contain every plausible word sequence. If a model assigns zero probability to every unseen continuation, a single missing sequence can make an otherwise plausible sentence receive zero probability. Add-k distributes some probability mass to those unseen alternatives.
The amount k controls the strength of this adjustment. Add-one, or Laplace smoothing, is the special case k = 1. Smaller positive values make a weaker adjustment for a fixed vocabulary and set of counts. The method is simple to explain, but simplicity does not establish that it is the best smoother for a particular task.
How it works
Let h be a context, w a possible next word, V the number of outcomes in the chosen vocabulary, and C(h,w) the observed count. Let N(h) be the sum of those continuation counts. Then P(w given h) = [C(h,w) + k] / [N(h) + kV].
Each outcome receives the same pseudocount, including outcomes already observed. The total added count is kV, which explains the denominator. Larger vocabularies therefore make the same k exert more overall influence.
Choose the vocabulary and handling of unknown words before applying the formula. A word outside that outcome set does not automatically receive a probability just because k is positive. If the entire context is unseen and k is positive, the formula returns a uniform distribution over the vocabulary.
Three practical examples
1. A teaching language model can use add-k to show why an unseen bigram should not necessarily be considered impossible, while keeping every calculation inspectable.
2. A count-based text classifier can smooth word counts within each class, reducing the chance that one absent training word makes a class likelihood collapse to zero.
3. A small intent-analysis experiment can compare different smoothing strengths on held-out examples. The choice should follow measured performance rather than assuming that k = 1 is universally appropriate.
Think of it this way
Think of giving every candidate a small starting allocation before distributing the observed votes. Candidates with no votes still receive something, while popular candidates keep most of their advantage. The analogy captures pseudocounts, but language-model probabilities are not election results. The starting allocation is a modeling choice, not evidence that the corresponding events actually occurred.
Worked example
Suppose an invented next-word vocabulary contains four outcomes: approve, review, reject, and defer. After one context, their counts are 8, 2, 0, and 0. The total count is 10. Without smoothing, the probabilities are 0.8, 0.2, 0, and 0.
With k = 0.5, the denominator becomes 10 + 0.5 times 4 = 12. The probabilities are 8.5/12, 2.5/12, 0.5/12, and 0.5/12: approximately 0.7083, 0.2083, 0.0417, and 0.0417. Their numerators sum to 12, so the exact probabilities sum to one.
The two unseen outcomes together receive 1/12, or about 8.33 percent. That mass has to come from the observed distribution. Interestingly, review rises slightly from 0.2 to about 0.2083, because uniform smoothing pulls probabilities toward 0.25; not every observed outcome necessarily decreases.
With the same counts but k = 1, the denominator is 14 and each unseen outcome receives 1/14, about 0.0714. Neither setting is declared better by this arithmetic alone. A held-out evaluation is needed. These counts and results are a teaching construction, not an experiment on customer data.
Illustrative case study
Fictional case: Birch Messages builds a transparent baseline classifier for routing short requests. Some classes have few examples, and missing word counts make the unsmoothed likelihoods brittle.
The team introduces add-k, chooses candidate k values using validation data, and reserves a separate test set for the final comparison. It documents the vocabulary and unknown-word policy with the model. The case illustrates an evaluation workflow; it reports no invented success rate and does not imply that smoothing fixes poorly labeled training data.
Common misconceptions and mistakes
Adding k only to the numerator breaks normalization. Another error is adding it only to zero counts while retaining the same formula; that defines a different adjustment and needs its own denominator.
Do not assume that every unseen sequence is reasonable. Add-k spreads equal pseudocounts across the specified outcomes without understanding their meaning. It also cannot recover missing domain knowledge or correct tokenization errors. For language modeling, more sophisticated smoothing methods can handle sparse contexts better; the textbook specifically warns that add-k is not generally a strong n-gram smoothing choice.
Business use
Add-k is useful as an auditable baseline and an explanation of how finite samples differ from complete knowledge. Its parameters and vocabulary are easy to document, which helps teams reproduce a count-based system.
When comparing models, use the same vocabulary and tokenization policy, and report the evaluation objective. A lower held-out language-model loss and a higher classification accuracy are different outcomes. Tune on validation data rather than repeatedly using the final test set to select k.
People also ask
Is k restricted to whole numbers?
No. Positive fractional values are common; the added quantities are pseudocounts rather than recorded occurrences.
What happens as k becomes very large?
For a fixed finite vocabulary and fixed observed counts, the probabilities approach a uniform distribution.
Does add-k solve unknown vocabulary?
Not by itself. The model needs an explicit outcome policy, such as a defined unknown-word token, to handle words outside its vocabulary.
Check your understanding
There are five possible outcomes, the observed total is 20, and k is 0.2. What is the probability of an outcome with zero observations?
Answer: 0.2 divided by (20 + 5 times 0.2), or 0.2/21, approximately 0.00952. Using 20 as the denominator would omit the added pseudocount mass.
Related terms
- Smoothing
- Laplace smoothing
- Pseudocounts
- N-gram
- Vocabulary
- Backoff
- pseudocount
- maximum likelihood estimation
- interpolation
Learn next
Compare add-k with interpolation and backoff on the same sparse count table. Examine what information a shorter context can supply that uniform pseudocounts cannot, and keep unknown-word handling explicit.
More in Natural language and speech
- Activity expressions - Activity expressions describe situations that unfold over time without building a particular completion point into their meaning.
- Adaptive softmax - Adaptive softmax is a hierarchical output-layer method for efficiently modeling large, unevenly used vocabularies.
- Add gate - An add gate is the learned control in an LSTM network that regulates how much newly proposed information enters its memory state.
- Adequacy - Adequacy is the degree to which a translation preserves the meaning of its source text.
- Adjacency pairs - Adjacency pairs are linked conversational actions in which a first contribution creates an expectation for a particular kind of response.
- Adjective - An adjective is a word class whose members typically express properties of nouns or the things those nouns refer to. In “a quiet room,” quiet modifies room.
Sources
Source: Jurafsky and Martin, Speech and Language Processing, Chapter 3, Section 3.6.2, read for add-k's formula, held-out parameter selection, and limitations for n-gram models: https://web.stanford.edu/~jurafsky/slp3/3.pdf . Calculations, examples, and the fictional case are original. The supplied full-book retrieval was unavailable; the accessible author-hosted chapter was used.