What does alpha zero do?

It makes positive-priority items equally likely to be sampled.

Why add a small positive priority offset?

To avoid making zero-error items permanently unsampleable under proportional prioritization.

Does beta one remove every source of learning bias?

No. It corrects the stated nonuniform replay sampling under the relevant assumptions, not all approximation or policy-distribution issues.

AI Glossary / Reinforcement learning and robotics / Intermediate

Prioritized experience replay

Definition

Prioritized experience replay is a reinforcement-learning technique that samples some stored experiences more often than others according to an estimate of their usefulness for learning. A common priority is the magnitude of a transition's temporal-difference error. Stochastic sampling preserves opportunities to revisit lower-priority experiences, while importance weights can compensate for the altered replay distribution.

What it means

Experience replay stores past transitions so a learner can use them again. Uniform replay gives every stored item the same sampling probability. Prioritized replay instead asks which items currently appear to offer more learning progress.

A large TD error means an observation differs substantially from the learner's current reward-plus-continuation prediction. Such an item may deserve more attention, but error magnitude is only a proxy. Noise, an inadequate model or a bad target can also produce persistent errors. Prioritization changes which stored data are used for updates; it does not by itself determine which experiences are collected or retained.

How it works

Assign each buffer item a positive priority p. In proportional prioritization, a common choice is absolute TD error plus a small positive constant. Sample item i with probability P(i) equal to its priority raised to alpha, divided by the sum of similarly transformed priorities. Alpha zero gives uniform sampling; larger alpha strengthens preference for high priorities.

Rank-based prioritization instead derives priority from the item's position when errors are sorted. This reduces sensitivity to the numerical magnitude of extreme errors, although it creates its own implementation choices.

Because prioritized sampling changes the average update, apply a correction weight such as (1 divided by N P(i)) raised to beta, where N is buffer size. Beta one gives the full correction toward a uniform-buffer expectation in the fixed-buffer calculation; beta zero gives no correction. Practical implementations may normalize weights and schedule beta over training.

After replaying an item, refresh its priority using its new error. Other priorities can become stale as the model changes. Efficient data structures help update and sample priorities without repeatedly scanning the entire buffer.

Three practical examples

1. A simulated agent revisits rare transitions with large prediction errors more often than redundant transitions it already predicts well.

2. A researcher compares uniform, proportional and rank-based replay using the same stored dataset and interaction budget.

3. A training audit investigates whether a few noisy experiences dominate updates despite producing little useful improvement.

Think of it this way

Think of a study schedule that gives difficult material more review time while occasionally revisiting familiar topics. Difficulty can indicate a valuable learning opportunity, but it can also indicate a confusing or incorrect exercise. A good schedule needs both prioritization and checks that repeated attention is actually helping.

Worked example

Consider an original fixed replay buffer with four items A, B, C and D. Their final positive priorities are 1, 2, 3 and 4; these values already include any positive offset used by the implementation. Set alpha to 1.

The priority sum is 10, so the sampling probabilities are 0.1, 0.2, 0.3 and 0.4. In 100 independent draws with replacement, the expected counts are 10, 20, 30 and 40. Actual counts vary, and a finite batch does not guarantee that every item appears.

Set beta to 1. Since N is 4, the unnormalized importance weights are 1/(4 times 0.1), 1/(4 times 0.2), 1/(4 times 0.3) and 1/(4 times 0.4). They are 2.5, 1.25, approximately 0.83333 and 0.625.

To check the correction, suppose the four items' scalar update contributions, before importance weighting, are 0, 0, 0 and 8. The uniform-buffer mean is 2. Under prioritized sampling without correction, the expected contribution is 0.4 times 8, or 3.2. With the unnormalized full correction, it is 0.4 times 0.625 times 8, or 2, recovering the uniform mean.

If all weights are divided by the buffer-wide maximum of 2.5, they become 1, 0.5, one-third and 0.25. The expected weighted contribution becomes 0.4 times 0.25 times 8, or 0.8. This is the uniform mean multiplied by the common scale factor 0.4, not an unchanged numerical expectation. That scaling can be absorbed into update-size interpretation, but should not be silently ignored. Batch-dependent normalization introduces additional details beyond this fixed calculation.

Illustrative case study

In a fictional project, Hazel Simulation uses prioritized replay for a sparse-feedback task. Training initially improves, but later a small group of transitions occupies a large fraction of sampled batches.

Investigation finds that these transitions contain unusually noisy rewards. The team compares rank-based sampling, less aggressive prioritization and uniform replay, while tracking coverage and held-out performance. It also checks priority refresh and correction weights. The project reports which changes help actual learning rather than assuming that repeatedly replayed high-error items are inherently important to the business objective.

Common misconceptions and mistakes

Priority is not the same as reward. A large negative error can receive high priority, and a high-reward transition may have low error after it is learned.

The method is not deterministic selection of the largest error. Stochastic variants deliberately retain lower-priority sampling opportunities.

Importance correction here addresses replay sampling relative to a chosen buffer distribution. It does not automatically solve every mismatch between behavior and target policies, missing coverage or obsolete data. Likewise, replay does not create information absent from the buffer. Excessive focus on noisy or stale items can reduce diversity and waste updates.

Business use

Prioritized replay can help when environment interactions are expensive and stored experience can be reused more effectively. Teams should compare interaction savings against memory, sampling overhead and tuning effort. Useful diagnostics include replay frequency, error persistence, correction weights and coverage of important situations. Evaluate task outcomes separately, because a priority measure tied to prediction error need not reflect operational value directly.

Check your understanding

Why is D sampled four times as often as A in expectation?

Answer: Its priority is four times larger and alpha is one.

Why does the normalized correction yield 0.8 rather than 2?

Answer: Dividing all weights by 2.5 scales the entire expected update by 0.4.

Related terms

Learn next

Next, recompute sampling probabilities with alpha equal to one-half. Then compare proportional priorities with rank priorities when one error becomes an extreme outlier.

Sources

Primary source: Tom Schaul and colleagues, Prioritized Experience Replay, Sections 3.2 through 3.4: https://arxiv.org/pdf/1511.05952

The four-item buffer, correction calculation and fictional case are original teaching material. The expectation check holds the buffer, probabilities and item contributions fixed, with explicitly stated normalization.

Previous: Prioritized circumscription Next: Prioritized sweeping