Does DDPG need an environment model?

It is model-free and learns from transitions rather than requiring a transition model.

Why use two versions of each network?

Slowly updated target networks provide more stable bootstrapping targets.

Does a larger critic value prove better behavior?

No. Independent evaluation is needed because the critic can be wrong.

AI Glossary / Reinforcement learning and robotics / Advanced

Deep deterministic policy gradient

Definition

Deep deterministic policy gradient, or DDPG, is an off-policy actor-critic reinforcement-learning algorithm for continuous action spaces. A neural-network actor maps a state to an action, while a critic estimates the expected return from a state-action pair. The actor learns by following the critic's gradient with respect to its proposed action.

What it means

Some decisions are numerical settings rather than choices from a short menu. A simulated controller might choose a continuous steering command or joint torque. DDPG learns an action-producing function instead of explicitly evaluating every possible numerical action.
Deterministic describes the actor's output for a given state and fixed parameters. It does not mean that the environment is deterministic or that training has no randomness. Experience collection commonly adds exploration noise to the actor's action, and sampling from stored experience also introduces randomness. The evaluated policy can be deterministic while its training behavior is exploratory.

How it works

The agent collects transitions containing the state, action, reward and next state and stores them in a replay buffer. Learning uses batches from this buffer, allowing experience collected by earlier or exploratory behavior to support the current update.
The critic is trained toward a bootstrapped target. For a nonterminal transition, the target has the form r + gamma Q_target(s_next, mu_target(s_next)). A genuine terminal transition uses no continuation value. Separate target actor and critic networks change slowly, helping prevent the prediction target from changing as abruptly as the networks being optimized.
The actor update uses the chain rule: the critic's derivative with respect to action is combined with the actor's derivative with respect to its parameters. This guides the actor toward actions that the critic predicts will have higher value. It does not require differentiating through the environment's physical dynamics. The critic can nevertheless be inaccurate, so improving its predicted objective is not a guarantee of improving actual return.

Three practical examples

Simulated steering: an actor produces a real-valued control signal while a critic learns from resulting trajectories. Evaluation examines complete driving behavior in the simulator, not just critic predictions.
Manipulation research: a policy outputs several continuous joint commands. The action representation and bounds must fit the simulated robot's control interface.
Algorithm comparison: researchers compare DDPG with alternatives under consistent training and evaluation conditions, checking variability across runs as well as average performance.

Think of it this way

Imagine a coach who suggests a numerical adjustment and an evaluator who predicts its likely result. The coach changes the suggestion in the direction the evaluator considers better. If the evaluator has learned the wrong relationship, the coach can become better at pleasing the evaluator without becoming better at the real task.

Worked example

Consider an actor update at one state. Let the actor's single parameter theta directly specify its action: mu_theta(s) = theta. Suppose the critic currently represents Q(s,a) = 5 - (a-2) squared. These are teaching functions, not trained neural networks or verified environment values.
At theta = 0.5, the actor proposes action 0.5. The critic predicts Q = 5 - 1.5 squared = 2.75. Its action derivative is -2(a-2), which equals 3 at that action. The actor derivative with respect to theta is 1, so the chain-rule gradient for this example is 3 times 1 = 3.
A gradient-ascent step with learning rate 0.1 gives theta_new = 0.5 + 0.1 times 3 = 0.8. The critic's prediction at the new action is 5 - 1.2 squared = 3.56. The predicted improvement is 0.81. This calculation demonstrates movement along a critic gradient; it does not demonstrate improved environment performance or a full DDPG iteration.
Separately, consider a nonterminal replay transition with reward 1, discount 0.9 and target-critic value 4 at the target actor's next action. The critic target is 1 + 0.9 times 4 = 4.6. If the current critic predicts 3 for the recorded state-action pair, the squared prediction error is (3-4.6) squared = 2.56. For a genuinely terminal transition with reward 1, the target would instead be 1.
Finally, consider a single target-network parameter currently equal to 0.6 while its corresponding online parameter is 0.8. With update convention target_new = (1-tau) target_old + tau online and tau = 0.05, the new target parameter is 0.95 times 0.6 + 0.05 times 0.8 = 0.61. Different software may name or define its averaging coefficient differently, so the equation matters more than the parameter label.

Illustrative case study

Fictional company Alder Control evaluates a DDPG policy in a simplified simulator. Its analysts record action bounds, exploration settings and target-update conventions. They compare returns from separate evaluation episodes with the critic's predictions rather than treating a falling training loss as the entire result.
If predicted values rise while evaluation deteriorates, the team investigates critic errors, data coverage and reward design. It also checks whether reported results depend heavily on a particular random run. This fictional workflow reports no measured results.

Common misconceptions and mistakes

DDPG is not simply DQN with continuous outputs. It uses a separate actor to propose actions and differentiates the critic through those actions. Its learned critic is a value estimator, not a learned transition model of the environment.
A deterministic actor does not remove the need for exploration. Replay also does not make every stored transition equally informative or eliminate distribution problems. Another mistake is updating target parameters with the wrong coefficient convention. Finally, neural function approximation and bootstrapping can be unstable; target networks and replay are useful mechanisms rather than universal convergence guarantees.

Business use

DDPG provides a reference method for research involving continuous sequential decisions. Evaluation should include reward design, simulation fidelity, data requirements and variability across training runs. A controller that performs well in a simplified simulator may need substantial further validation before it supports an operational task. Algorithm selection should follow task evidence.

Check your understanding

Why did the example actor move from 0.5 to 0.8?

Answer: It took a 0.1-sized step along gradient 3.

Why was the nonterminal target 4.6?

Answer: It added reward 1 to discounted continuation 0.9 times 4.

Why was the target parameter updated only to 0.61?

Answer: The averaging rule retained most of its previous value.

Related terms

Deterministic policy gradient
Replay buffer
Target network
Actor-critic
Continuous action space
TD3
Exploration noise

Learn next

Trace one replay transition through the critic target, then distinguish the actor gradient from the critic's prediction-loss update.

Sources

Lillicrap and colleagues, Continuous Control with Deep Reinforcement Learning, ICLR 2016, DDPG formulation: https://arxiv.org/pdf/1509.02971 . The scalar calculations and company scenario are original illustrations.

Previous: Deep Copy Next: Deep ensemble