Does Double DQN average two estimates?

No; one network selects an action and the other evaluates it.

Are the networks independent?

No; target updates connect their parameters.

Is its target always more accurate?

No. Reduced selection-related overestimation is not a per-transition accuracy guarantee.

AI Glossary / Reinforcement learning and robotics / Advanced

Double deep Q-network

Definition

A double deep Q-network, usually called Double DQN, is a reinforcement-learning method that separates choosing a promising next action from estimating that action's value when constructing a training target. It modifies the target used by a deep Q-network to reduce overestimation associated with maximizing noisy value estimates. It still learns an action-value function for discrete actions.

What it means

A value estimate can be high because an action is genuinely good or because estimation error happens to favor it. If the same estimates both select the highest-valued action and supply its estimated value, positive errors can disproportionately influence the result.
Double DQN reduces this coupling. The current online network chooses the next action, while a target network evaluates that chosen action. This is a change to learning targets, not a promise that all learned values become unbiased or that every training run improves. The networks remain related because target-network parameters are periodically copied from the online network.

How it works

Training uses transitions containing a state, action, reward and next state, commonly sampled from replay memory. For a nonterminal transition, the online network first selects a_star = argmax_a Q_online(next_state,a). The target is then reward + gamma times Q_target(next_state,a_star).
Ordinary DQN instead uses reward + gamma times max_a Q_target(next_state,a). The difference becomes visible when the networks disagree about which next action has the highest value. Double DQN does not ask the target network to select its own favorite action for that target.
The online network is trained to bring its estimate for the recorded state-action pair closer to the target. The target network remains fixed between its scheduled updates in the original method. A true terminal transition has no future-value bootstrap. Handling a data cutoff that is not an actual terminal state requires a separate decision about the task and available information.

Three practical examples

Game simulation: a discrete-action agent learns from stored play transitions while separating action selection from target evaluation.
Scheduling simulation: an agent chooses among a finite set of dispatch rules, with Double DQN supplying the value-learning update rather than determining the operational reward definition.
Algorithm comparison: researchers compare DQN and Double DQN with otherwise matched settings to study value estimates and policy performance. Multiple runs help distinguish a systematic effect from random variation.

Think of it this way

Imagine one assessor nominates the strongest proposal, while another supplies the score used for that nominee. The second assessor does not replace the nominee with a different favorite. This analogy illustrates the separation, although the two networks in Double DQN are related rather than fully independent assessors.

Worked example

Consider an original nonterminal transition with reward 2 and discount factor 0.9. At its next state, three available actions are A, B and C. The online network estimates their values as 8, 10 and 6. The target network estimates them as 12, 7 and 9.
The online network selects B because 10 is its highest estimate. Double DQN then reads the target network's value for B, which is 7. Its training target is 2 + 0.9 times 7 = 8.3.
Ordinary DQN would take the target network's maximum, 12 for A, producing 2 + 0.9 times 12 = 12.8. The targets differ by 4.5. Double DQN does not average 10 and 7, take the smaller estimate for every action, or use the online value 10 as the bootstrap value.
Suppose the online network currently predicts 9 for the recorded state-action pair. Its Double DQN target-minus-prediction error is 8.3 - 9 = -0.7, and the squared error is 0.49. Against the ordinary DQN target, the error would be 12.8 - 9 = 3.8, with squared error 14.44. These are illustrative loss calculations, not the exact parameter changes made by a neural-network optimizer.
If the transition instead ends the task, both methods use the immediate reward 2 as the target. With prediction 9, the error becomes -7. If the two networks agree on the maximizing next action, their DQN and Double DQN bootstrap targets also agree for that transition.
Nothing in the supplied estimates establishes the true values of A, B or C. Consequently, the smaller Double DQN target in this example does not by itself prove greater accuracy. The example demonstrates which estimate enters the calculation; accuracy requires evidence about actual expected returns.

Illustrative case study

Fictional company Birch Simulation studies agents that select among discrete warehouse scheduling rules. Its team implements both target calculations and checks them against small hand-calculated transitions before comparing training runs.
The analysts record achieved reward alongside value estimates. If predicted values fall under Double DQN, they investigate whether this corresponds to better calibration and better decisions rather than treating lower values as success automatically. They keep reward definitions and evaluation conditions consistent. This fictional scenario reports no operational outcome or measured performance gain.

Common misconceptions and mistakes

Double DQN does not require adding two more networks to ordinary DQN: it reuses DQN's online and target networks with different roles in the target. Nor is it the same as clipped double Q-learning, which uses a minimum of critic estimates in methods such as TD3.
Another mistake is letting the target network both select and evaluate the next action while calling the result Double DQN. That is the ordinary DQN bootstrap calculation. Finally, reducing one source of overestimation does not resolve inadequate exploration, poor reward design or missing coverage in training data.

Business use

Double DQN is relevant when a simulated decision problem has discrete actions and a team is evaluating value-based reinforcement learning. It provides a focused algorithmic comparison. A useful report distinguishes estimated values, realized evaluation returns and variability across runs, and documents terminal handling and target-update settings. Algorithm choice alone does not establish suitability for deployment.

Check your understanding

Why was B selected?

Answer: It had the highest online-network value.

Why was 7 used?

Answer: It was the target network's estimate for B.

Why could accuracy not be established?

Answer: The example supplied estimates, not true action values.

Related terms

Learn next

Construct another pair of value tables and compare targets when the maximizing actions agree and disagree.

Sources

Van Hasselt, Guez and Silver, Deep Reinforcement Learning with Double Q-learning, Double DQN section: https://arxiv.org/pdf/1509.06461 . The numerical tables and company scenario are original illustrations.

Previous: Dot-product attention Next: Double delta feature