Are the two critics averaged?

Not in the clipped target; their minimum is used.

Does the method guarantee underestimation?

No. Both critics can overestimate the same action.

Is this the whole TD3 algorithm?

No. TD3 includes additional mechanisms.

AI Glossary / Reinforcement learning and robotics / Advanced

Clipped double-Q learning

Definition

Clipped double-Q learning uses the smaller of two action-value estimates when constructing a learning target. Introduced as a component of the TD3 actor-critic method, it aims to limit overestimation caused by function-approximation errors. The clipping refers to selecting the lower critic estimate, not clipping rewards to a fixed numerical interval.

What it means

A policy can favor actions whose predicted values are high partly because of estimation error. If the learner repeatedly treats those optimistic predictions as reliable targets, the error can influence both value learning and action selection.
Two critics provide two estimates for the same state and candidate action. Taking their minimum makes the target no larger than either individual estimate for that comparison. This is a deliberately cautious construction, but it does not make the target a guaranteed lower bound on the true value. Both critics can be wrong in the same direction, and selecting a minimum can also create underestimation.

How it works

In a common TD3 formulation, a transition supplies reward r, next state s_next and a terminal indicator d. A target actor produces a next action, with target-policy smoothing applied as specified by TD3. Both target critics evaluate that same next-state action.
The target is y = r + gamma (1-d) min(Q1_target(s_next, a_next), Q2_target(s_next, a_next)). Both current critics learn toward this target through their respective prediction losses. A genuine terminal transition has no bootstrapped continuation term. A time-limit truncation requires interpretation according to the task rather than automatically being treated as terminal.
TD3 also includes delayed policy updates and target-policy smoothing. Those mechanisms are separate from the minimum operation. Its actor update uses the first critic in the original implementation, rather than simply taking the minimum everywhere. Two networks trained from related data are not automatically independent; shared experience and targets can correlate their errors.

Three practical examples

Continuous-control research: a simulated agent uses two critics to construct less optimistic bootstrapped targets while learning a deterministic policy.
Algorithm comparison: researchers remove the minimum operation while keeping other TD3 components fixed, then evaluate how value estimates and policy performance change.
Diagnostic analysis: a team examines critic disagreement on sampled actions. Disagreement can reveal differing predictions, but it is not by itself a calibrated uncertainty interval or proof that the lower estimate is correct.

Think of it this way

Imagine receiving two estimates for the value of a project and using the lower one in a planning calculation. This avoids relying on the more optimistic estimate, but both estimates may share a mistaken assumption. The procedure is cautious in a specific arithmetic sense; it does not certify the project's true value.

Worked example

Consider an original nonterminal transition with reward 2 and discount gamma = 0.9. Suppose two target critics evaluate the same next action as 8 and 12. The clipped continuation estimate is min(8,12) = 8, so the learning target is 2 + 0.9 times 8 = 9.2.
Using the larger estimate would give 2 + 0.9 times 12 = 12.8. Averaging the two estimates would give 2 + 0.9 times 10 = 11. These are different target constructions. Clipped double-Q learning uses the minimum in this example, not the average. If the transition were genuinely terminal, the target would instead be the immediate reward 2.
Now examine a separate toy error model to understand bias. Suppose the true continuation value is 10, and each critic independently reports either 8 or 12 with equal probability. Each critic is individually unbiased because its average is 10.
The four equally likely pairs are (8,8), (8,12), (12,8) and (12,12). Their minima are 8, 8, 8 and 12. The expected minimum is therefore (8+8+8+12)/4 = 9, one unit below the true value. With the same reward and discount, the expected clipped target is 2 + 0.9 times 9 = 10.1, compared with the true one-step target 2 + 0.9 times 10 = 11.
If the two critics instead always make identical paired errors, only (8,8) and (12,12) occur, each half the time. The expected minimum is then 10. This comparison illustrates why error dependence matters. It does not claim that trained critics follow either toy distribution.
Finally, if both critics report 12 when the true continuation value is 10, taking the minimum still gives 12. The rule cannot remove shared overestimation merely by selecting the smaller of identical values. All values here are stipulated arithmetic examples, not results from a training experiment.

Illustrative case study

Fictional company Maple Control compares simulated policies trained with and without clipped targets. Its analysts monitor estimated values, independent return estimates and task performance across multiple runs. They retain the same reward definition and distinguish the minimum operation from changes to policy-update timing.
When critics disagree less, the team checks whether their predictions improved or simply became more correlated. It does not label smaller disagreement as proof of accuracy. This fictional evaluation reports no deployment or measured performance gain.

Common misconceptions and mistakes

Clipped double-Q learning is not identical to conventional Double Q-learning, which separates action selection and evaluation using different estimates. It also does not mean clipping gradients, rewards or policy probability ratios.
The minimum is not a guaranteed confidence bound and does not eliminate every source of bias. The two estimates should refer to the intended same target action; independently maximizing each critic and then taking a minimum describes a different calculation. Finally, benefits of the complete TD3 algorithm cannot automatically be attributed entirely to this one component without an appropriate comparison.

Business use

The method is relevant when evaluating actor-critic systems for sequential control research. Teams should assess actual policy outcomes and value-estimation behavior, not just whether targets became smaller. A more pessimistic target can reduce one error while creating another. Comparisons should hold simulation budgets and other algorithm settings sufficiently consistent to support interpretation.

Check your understanding

Why was the first target 9.2?

Answer: It used reward 2 plus 0.9 times the lower continuation estimate 8.

Why did unbiased toy critics produce a biased minimum?

Answer: Selecting the minimum favored their negative errors.

Why can correlated errors matter?

Answer: Both critics may repeat the same mistake, limiting the benefit of comparing them.

Related terms

Double Q-learning
Critic
Target network
Target policy smoothing
TD3
Overestimation bias
Actor-critic

Learn next

Compare minimum, average and single-critic targets on fixed transitions, then evaluate how each affects complete policy training.

Critic

Sources

Fujimoto, van Hoof and Meger, Addressing Function Approximation Error in Actor-Critic Methods, ICML 2018, Section 4.2 and TD3: https://proceedings.mlr.press/v80/fujimoto18a/fujimoto18a.pdf . The target and bias calculations and company scenario are original illustrations.

Previous: Clinical trial Next: Clipping