Why use the smaller critic estimate?

To reduce the effect of optimistic approximation errors in the bootstrapped target.

Does the actor use both critics in its original update?

The original actor gradient uses the first critic.

Can TD3 learn from earlier policies' transitions?

Yes, through its off-policy replay formulation, subject to data quality and coverage limitations.

AI Glossary / Reinforcement learning and robotics / Advanced

Twin delayed deep deterministic policy gradient

Definition

Twin delayed deep deterministic policy gradient, commonly abbreviated TD3, is an off-policy actor-critic algorithm for continuous-action reinforcement learning. It modifies deterministic actor-critic learning through three central techniques: twin critics with a minimum-based target, delayed actor updates, and target policy smoothing. Together, these techniques address errors in approximate action values that can mislead policy learning.

What it means

A deterministic actor maps a state to one action. A critic estimates the long-term return from taking an action and then following the policy. When the critic has an inaccurate high-value peak, the actor may exploit that error rather than discover genuinely better behavior.

TD3 learns two critic functions and uses the lower target estimate during bootstrapping. It also lets the critics receive more updates than the actor, and perturbs the target action so the learning target reflects nearby actions.

Twin means two critics, not two competing actors. Delayed means the actor is updated less frequently than the critics, not that rewards or environment actions are intentionally delayed. The actor remains deterministic even though exploration and target construction can involve randomness.

How it works

Store transitions in a replay buffer and sample minibatches. For each nonterminal next state, compute a target-actor action, add clipped Gaussian noise and clip the result to valid action bounds. Evaluate the same perturbed action using both target critics.

Form the target as immediate reward plus gamma times the smaller target-critic value. Fit both current critics to this detached target. Genuine terminal transitions use only immediate reward.

Every specified number of critic updates, update the actor to increase the first current critic's estimate for the actor's own actions. The original TD3 actor update uses that first critic, rather than automatically differentiating the minimum of the two. Update the target actor and target critics slowly on the delayed schedule.

During data collection, add a separately configured exploration perturbation to the current actor's action. This differs from target smoothing, which affects learning targets. Replay reuse makes the method off-policy, but inadequate coverage, correlated critic errors and inappropriate noise scales can still impair learning. These practical techniques do not provide unconditional convergence or eliminate all overestimation.

Three practical examples

1. A simulated continuous controller uses two critics to reduce dependence on a single optimistic continuation estimate.

2. A research team checks whether fewer actor updates help when policy changes otherwise outrun critic learning.

3. An implementation audit verifies that exploration noise, target noise and moving-average target updates are treated as separate mechanisms.

Think of it this way

Think of a planner consulting two appraisals, using the more cautious appraisal for future planning, and waiting for additional appraisal work before revising the plan. The planner also checks a small neighborhood around the proposed action. This can reduce sensitivity to a fragile estimate, but two appraisers can still share the same mistake.

Worked example

Consider an original scalar-action target calculation. Actions must lie between -1 and 1. The target actor outputs 0.4 at the sampled next state. A target-noise draw of -0.3 is clipped to -0.2, producing a valid perturbed action of 0.2.

Suppose the target critics at that action output 5 and 4. The observed reward is -1 and gamma is 0.95. The target is -1 plus 0.95 times the smaller estimate 4, or 2.8. Averaging the target critics would instead give 3.275, which is not the stated TD3 target.

If the current critics predict 3 and 2 for the recorded state-action pair, their squared errors against 2.8 are 0.04 and 0.64. Their summed single-sample critic loss is 0.68. These specify an objective contribution, not weight updates, which also depend on gradients and the optimizer.

Now consider an update-frequency setting of two. Across six critic-training iterations, each critic is updated six times, while the actor and target networks are updated on iterations 2, 4 and 6: three times each. These are training iterations, not environment steps.

For a separate target-parameter illustration, let a target weight be 2, its corresponding newly updated online weight be 3, and the moving-average coefficient be 0.1. The target becomes 0.1 times 3 plus 0.9 times 2, or 2.1. This parameter averaging is distinct from adding noise to an action.

Illustrative case study

In a fictional project, Willow Actuation compares TD3 with a simpler deterministic actor-critic in a simulated positioning task. The team uses equal interaction budgets and records multiple runs rather than selecting only the best seed.

A review finds that actor updates were delayed correctly, but target networks were updated on every critic step. Researchers align the schedule with the intended implementation before interpreting the comparison. They also report the noise scales relative to normalized action units.

This illustrative case concerns reproducibility, not a deployment or measured gain. Movement limits and failure handling remain separately specified requirements.

Common misconceptions and mistakes

Do not replace the twin minimum with an average while claiming the same algorithm. Do not assume two differently initialized critics have statistically independent errors after training on shared data. Do not confuse delayed policy updates with a guarantee that the critics have fully converged before each actor update.

Also avoid describing TD3 as entropy-regularized learning. Unlike SAC, the standard TD3 objective does not add a policy-entropy bonus. Its stochastic perturbations serve exploration and target regularization around a deterministic actor.

Business use

TD3 is a useful baseline for continuous-control experiments when replay reuse and a deterministic policy are appropriate. Assess actual task return, failure rates, interaction cost and sensitivity to settings. Record critic-update counts, actor frequency, target coefficients, action scaling and both noise definitions so results can be reproduced. A low critic loss alone does not establish that the learned actions are operationally effective.

Check your understanding

What target is used in the example?

Answer: 2.8.

How many actor updates occur in six training iterations at frequency two?

Answer: Three.

Why does a target weight move from 2 to 2.1 rather than 3?

Answer: The moving-average update retains 90 percent of its previous value.

Related terms

Learn next

Study deterministic policy gradients and critic bootstrapping first. Then isolate each TD3 modification in small tests before combining them. Compare TD3 and SAC through their objectives and action distributions, not only through the fact that both use twin critics.

Sources

Fujimoto, van Hoof and Meger, Addressing Function Approximation Error in Actor-Critic Methods, ICML 2018: https://proceedings.mlr.press/v80/fujimoto18a/fujimoto18a.pdf.

Author implementation for target construction and delayed schedules: https://github.com/sfujim/TD3/blob/master/TD3.py. The calculations and fictional case are original illustrations.

Previous: Tweedie's formula Next: Two-filter smoothing