Why does correlation between consecutive steps matter?

Training on correlated streaks makes updates chase the recent past in circles instead of converging on general lessons.

What size buffer is typical?

From hundreds of thousands to millions of records, depending on how fast the environment changes and how much history helps.

Is replay used outside reinforcement learning?

The same idea, revisiting shuffled past examples, appears wherever continual learning must avoid forgetting.

AI Glossary / AI and machine learning / Intermediate

Experience replay

Definition

Experience replay is a training technique where a reinforcement learning agent stores its past experiences and repeatedly trains on random samples of them, instead of learning only from its most recent actions.

What it means

An agent learning online sees its experiences in order, and consecutive experiences are highly correlated: one minute of a delivery run looks much like the next. Training on such streaks sends learning in circles, endlessly adjusting to the latest moment.

Experience replay breaks the streaks. Every experience, the situation, the action, the reward, the next situation, goes into a memory buffer, and each training step draws a random batch from the whole buffer.

Random sampling decorrelates the training data, reuses each experience many times, and smooths learning over the agent's whole history rather than its latest episode.

The buffer is finite, so old experiences are pushed out; what the agent can learn from is always a rolling window on its past. Very old lessons can be forgotten as thoroughly as in any organization.

For a business, experience replay is the reason reinforcement learning became practical: it turned a trickle of sequential experience into a reusable training asset, the difference between learning from each mistake once and learning from it a hundred times.

Variants refine the basic scheme: prioritized replay samples informative experiences more often than routine ones, so the agent studies its surprises harder than its successes.

How it works

Store every step as a record: the situation observed, the action taken, the reward received and the situation that followed.

Add each record to a fixed-size buffer, discarding the oldest when full.

At each training step, draw a random batch of records from the buffer, not the most recent records.

Train the value network on that batch, so updates reflect a shuffled cross-section of the agent's history.

Continue acting, storing and sampling; the buffer grows more representative as the agent explores.

Watch the buffer's composition as the world drifts: if old regime experiences dominate, the agent keeps learning yesterday's lessons, and shrinking or refreshing the buffer becomes the fix.

Three practical examples

Game agents: replaying past frames so learning is not dominated by the current level's quirks.
Pricing agents: retraining nightly on shuffled records from months of price experiments.
Robotics: replaying stored grasps so a rare successful lift is learned from many times.

Think of it this way

Think of a trainee who reviews a shuffled stack of past case files every evening rather than only today's. Today's cases are fresh but repetitive; the shuffled stack covers quiet Tuesdays and crisis Fridays alike, and the trainee's judgment stops swinging with the day's weather.

The limit: the stack ages. Cases from before the reorganization teach obsolete lessons, which is why the buffer forgets the oldest files.

Worked example

stabilizing a delivery agent

Suppose a delivery-routing agent trained online improves for a week, then suddenly unlearns everything after a snowstorm dominates its recent experience. The figures are illustrative.

With a replay buffer of one million steps, each training batch mixes storm days with ordinary days in their true proportions.

After the next storm, performance dips slightly instead of collapsing, because no single week can dominate the shuffled training data.

The operations team adds buffer size and sampling mix to their model review checklist.

Illustrative case study

A fictional retailer trained a markdown agent purely online and found it chasing whatever happened last week: one viral weekend permanently distorted its pricing.

Replay with a month-long buffer ended the whiplash, but also revealed a new problem: lessons from a discontinued product line lingered for weeks.

The invented lesson: replay makes learning stable and memory long. Choose the buffer's length as deliberately as you choose the agent's goals.

Common misconceptions and mistakes

Replay is just saving the data. It is a specific training scheme: storing steps and training on random batches, which changes what the agent learns, not just what it keeps.
Bigger buffers are always better. Bigger buffers remember longer but adapt slower to a changed world; the trade is real.
It removes the need for exploration. Replay reuses what exploration produced; it does not replace the exploring.

Business use

Ask how any reinforcement learning system trains between events: replay is the standard answer, and its absence is a red flag for instability.

Treat buffer design as policy: its size sets the agent's memory horizon, which sets how fast it adapts to regime change.

Log buffer statistics in monitoring; a buffer silently full of one week's data is a model about to surprise you.

When scoping a reinforcement learning project, ask how much experience the buffer will hold and how old the oldest record can be; the answer is the agent's effective memory policy.

Check your understanding

What two problems does experience replay solve for an agent trained on its own recent actions?

Answer: It breaks the correlation between consecutive experiences that destabilises learning, and it lets each experience be learned from many times instead of once.

Related terms

Learn next

AI Agents and Automation for Small Business explains the training and monitoring disciplines that keep automated agents stable: https://aiforbusinessandlife.com/academy/ai-agents-automation-small-business. [C4]

Sources

[S2] Google, Machine Learning Glossary. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Expectiminimax value Next: Experiment