Does a cache stampede require malicious traffic?

No. Normal simultaneous demand after expiry or a cold start can trigger it.

Is serving stale data always the solution?

No. It is suitable only where the application's freshness and permission requirements allow it.

Does probabilistic refresh guarantee one source request?

No. It manages probabilities and expected load; there may be no refresher or several, depending on the design and arrivals.

AI Glossary / Data engineering and architecture / Intermediate

Cache stampede

Definition

A cache stampede occurs when many requests encounter the same missing or expired cached value and independently try to fetch or recompute it. The resulting burst of duplicate work can overload the underlying service that the cache was intended to protect.

What it means

A cache stores reusable results so that later requests can avoid repeating expensive work. When a popular entry disappears, many callers may notice the miss before the first replacement has finished. Each may start the same database query, rendering task or computation.

This is also called dogpiling or a thundering-herd problem. It can happen during ordinary traffic, without an attack. Expiry, eviction, a cache restart or a deployment with empty local caches can trigger it.

A high average cache-hit rate does not rule out the problem. A brief concentration of misses on one expensive item can cause severe disruption while most other requests still hit cached values. The timing and cost of misses matter alongside their total number.

How it works

In a simple cache-aside flow, each request checks for a value and fetches it from a source if absent. Without coordination, concurrent callers all follow the miss path. If the source slows under the extra work, more requests arrive before a replacement is ready, extending the vulnerable period.

Request coalescing lets callers share an in-progress fetch for the same key. Depending on the design, other callers wait, receive an acceptable older value or get a controlled fallback. The scope matters: coordination within one process does not automatically coordinate an entire fleet.

Other approaches include controlled refresh before expiry and randomized timing. Spreading expiry across different keys can reduce synchronized waves, but does not by itself prevent duplicate work for one hot key. Probabilistic early refresh reduces expected contention without guaranteeing exactly one refresher. Each approach has tradeoffs involving freshness, latency and additional coordination dependencies.

Three practical examples

A popular product page expires just before a busy period, causing many servers to run the same expensive query.
A newly deployed service fleet starts with empty local caches and sends a burst of repeated requests to a shared dependency.
An AI application repeatedly recomputes the same permitted, reusable document representation because simultaneous misses are not coalesced.

Think of it this way

Think of a reception desk that normally displays a printed timetable. When the timetable is removed for updating, every visitor independently calls the timetable office for the same answer. The office becomes overwhelmed with duplicate requests.

One person obtaining an update while others wait can reduce that duplication. The analogy does not decide whether an old timetable is safe to display. In software, serving stale data must be an explicit decision based on the information and its use, not an automatic response to overload.

Worked example

Suppose a fictional service receives 300 requests per second for one cached report. Rebuilding the report takes 0.4 seconds when the source is healthy. If the entry expires and every request starts its own rebuild until the first result becomes available, roughly 300 times 0.4, or 120 requests, can enter the miss path during that interval.

That is an illustrative estimate under steady arrivals and unchanged rebuild time, not a measured peak. If duplicate rebuilds slow the source, the window can become longer and the number larger.

With correctly scoped single-flight coordination, one rebuild can serve the waiting requests for that key during the refresh. But if coordination exists independently in each of ten application processes, the fleet might still perform ten concurrent rebuilds. A design review must therefore identify the actual sharing boundary rather than simply record that “a lock exists”.

Illustrative case study

Illustrative case study: A fictional events directory sees short database overloads even though its daily cache-hit rate looks excellent. The team correlates the spikes with expiry of its most popular venue summary. It introduces per-key request coalescing and an approved, bounded stale-response window for that summary.

Testing includes a failed refresher, slow source responses and simultaneous requests from several service instances. The team also checks that unrelated keys can refresh independently. It monitors duplicate source calls and refresh duration, not just cache-hit percentage.

This invented case illustrates diagnosis and bounded mitigation. It makes no claim about a real service's improvement. The stale-response policy is specific to the fictional summary; it should not be assumed appropriate for permissions, balances or other freshness-sensitive information.

Common misconceptions and mistakes

Assuming a good average hit rate guarantees a healthy miss path.
Believing a longer time-to-live eliminates the eventual expiry problem.
Using a process-local lock while assuming fleet-wide coordination.
Treating expiry jitter across keys as complete protection for one popular key.
Allowing waiting callers to retry without bounds when a refresh fails.

Business use

For businesses, stampedes matter because a performance optimization can become a source of concentrated dependency load. Test cold starts, cache loss and popular-item expiry before relying on the cache to protect a smaller backend.

Define freshness requirements and failure behavior for each important class of data. Measure refresh concurrency, source latency and the number of callers waiting for one result. For AI applications, confirm that a cached result is genuinely reusable across the relevant users and permissions. Coalescing unrelated or differently authorized requests can create a correctness or access-control problem even if it reduces computation.

Check your understanding

Ten independent processes each allow one refresh per key. Does that guarantee only one refresh across the whole service?

Answer: No. The processes can each initiate a refresh, giving up to ten concurrent refreshes under the stated arrangement. Coordination scope must match the intended guarantee.

Related terms

Cache miss
Cache-aside
Time to live
Request coalescing
Single flight
Stale-while-revalidate
Thundering herd
Backpressure
Cache invalidation

Learn next

Trace two simultaneous requests through a cache miss. Then add a shared in-progress result and define what happens if its producer fails. Test expiry, empty-cache startup and coordination across multiple instances.

Sources

Primary sources: Amazon Builders' Library, Caching challenges and strategies, especially request coalescing and cold-cache behaviour: https://aws.amazon.com/builders-library/caching-challenges-and-strategies/ ; Cloudflare, Sometimes I cache, cache locks and probabilistic revalidation: https://blog.cloudflare.com/sometimes-i-cache/ . Relevant returned sections reviewed. Request counts and events-directory scenario are original illustrations, not vendor benchmarks.

Previous: Cache-only memory architecture Next: Calculus of variations