Does every cached word become more probable?

No. In a weighted mixture, its probability rises only if the cache component assigns it more probability than the base component does.

Is this the same as a transformer KV cache?

No. A KV cache reuses intermediate attention data. A predictive language-model cache adds an adaptive memory-based component to the output distribution.

AI Glossary / Natural language and speech / Intermediate

Cache language model

Definition

Definition: A cache language model adapts word or token predictions using a memory of previously encountered text. A cache component gives additional influence to relevant earlier occurrences and is commonly combined with a broader language model.

What it means

What it means: A document often repeats its names, specialist terms and topics. A general model may assign low probability to a rare product name even after that name has appeared repeatedly in the current document. A language-model cache offers a way to use this local evidence.
Here, cache means predictive memory. It is different from storing a completed response for reuse or retaining transformer key and value tensors to avoid repeated computation. Those engineering caches can accelerate calculation without introducing the separate adaptive probability distribution described here. Terminology matters when comparing claims about prediction quality, memory use and serving speed.

How it works

How it works: A simple count-based version stores recent tokens, estimates their relative frequencies and mixes that distribution with the base model's prediction. A window, reset policy or decay rule controls which history remains influential. The mixture weight determines how strongly the recent evidence affects the result.
A continuous neural cache instead stores contextual vectors together with the tokens that followed those contexts. At prediction time, similarity to stored contexts weights their associated tokens. Extensions can search larger memories rather than only a short recent window. These choices are model designs, not interchangeable implementation details. A system must also define its vocabulary, empty-cache behavior and whether observations or generated predictions enter the memory.

Three practical examples

Three practical examples:
1. A document-completion prototype adapts to repeated project terminology within the current draft. Its evaluation checks whether suggestions remain useful when the writer switches to a different subject.
2. A speech-recognition experiment uses recent accepted words to adjust language-model preferences. Researchers examine whether a mistaken transcription contaminates later predictions through the cache.
3. A research system compares frequency-based memory with contextual retrieval. The comparison isolates the cache contribution while holding the base model and evaluation sequence constant, rather than attributing every performance difference to memory.

Think of it this way

Think of it: A cache language model resembles keeping a temporary note of the vocabulary and contexts encountered during a discussion. That note can help anticipate what comes next. The analogy is limited because the cache records statistical evidence, not understanding: repetition does not establish truth, relevance or permission to reuse information in another conversation.

Worked example

Worked example: Use a toy vocabulary containing only valve, pump and other. At a particular prediction point, the base model assigns probabilities 0.10, 0.20 and 0.70. The previous ten observed tokens contain four occurrences of valve, one of pump and five of other. The count cache therefore assigns 0.40, 0.10 and 0.50.
Choose a cache weight of 0.25 and a base weight of 0.75. The mixed probability of valve is 0.75 × 0.10 + 0.25 × 0.40 = 0.175. For pump it is 0.75 × 0.20 + 0.25 × 0.10 = 0.175. For other it is 0.75 × 0.70 + 0.25 × 0.50 = 0.650. The three probabilities sum to one.
Valve gains probability while pump loses some, despite both having appeared in the cache. Appearing in memory does not guarantee an increase; the cache probability must exceed the base probability for this mixture to raise it. The calculation is an original count-based illustration, not a reconstruction of the continuous-cache experiments.

Illustrative case study

Illustrative case study: Fictional Beacon Notes evaluates a cache-assisted completion tool on equipment reports. It initially carries recent vocabulary across report boundaries. Reviewers observe suggestions associated with the preceding report after the user opens an unrelated document.
The team tests resetting the cache at document boundaries and compares the effect with retaining it within a continuing report. It also measures performance on first mentions, repeated terms and topic changes separately. This invented case illustrates the importance of a memory policy. It does not claim that resetting always wins or that a cache removes the need for a well-trained base model.

Common misconceptions and mistakes

Common misconceptions and mistakes: A larger cache is not automatically better; stale or irrelevant material can distort predictions. Updating a memory is not necessarily the same as changing neural-network weights. A cache also does not automatically support arbitrary new output tokens: vocabulary and copying mechanisms determine that capability. Finally, evaluating a next-token predictor with future text already placed in its memory leaks the answer. The cache must contain only the history permitted at each prediction point.

Business use

Business use: Ask whether a proposed cache changes predictions, accelerates computation or does both. For predictive adaptation, evaluate documents in the order the system will encounter them and document reset rules. Measure task quality alongside memory cost and latency. Where histories belong to different users or projects, define the intended separation and retention behavior. Consider how corrections update stored evidence. A cache built from mistaken generated text can reinforce a local error, so evaluation should reflect the actual source of cache entries.

Check your understanding

Check your understanding: The base probability of a token is 0.30 and its cache probability is 0.10. With equal mixture weights, what is the result?

Answer: 0.20. The cache lowers the token's probability in this example, even though it has a nonzero cache frequency.

Related terms

Perplexity
Domain adaptation
KV cache
adaptive language modelling
neural cache
interpolation
contextual representation
nearest-neighbour language model

Learn next

repeat the toy mixture with cache weights of zero, one-half and one. Then simulate a topic change by replacing the cache counts. Explain the difference between changing the prediction distribution and merely computing the same distribution more efficiently.

Sources

Sources: Grave, Joulin and Usunier, Improving Neural Language Models with a Continuous Cache, preprint: https://arxiv.org/pdf/1612.04426 . Grave, Cisse and Joulin, Unbounded Cache Model for Online Language Modeling with Open Vocabulary, 2017: https://arxiv.org/pdf/1711.02604 . Relevant primary preprint text was read; OpenReview's PDF endpoint returned a browser-verification page. The probability calculation and Beacon Notes case are original illustrations.

Previous: Cache coherence Next: Cache-only memory architecture