Is this model-free learning?

Not in the formulation here; it explicitly learns an environment model.

Does it automatically find an optimal policy?

The passive version evaluates the supplied policy.

Must it use a neural network?

No. The teaching formulation can use tables.

AI Glossary / Governance, ethics and safety / Advanced

Adaptive dynamic programming

Definition

Adaptive dynamic programming is a model-based reinforcement-learning approach that learns how an environment behaves from experience and uses dynamic programming to estimate long-term outcomes. In the AIMA treatment followed here, passive adaptive dynamic programming evaluates a fixed policy using a learned transition and reward model. Passive means the policy is given, not that the system does no learning.

What it means

Imagine following a warehouse routing rule whose consequences are not yet known. Observations reveal how often a corridor remains clear, becomes blocked or leads to the destination. A model records those patterns; a planning calculation estimates the accumulated reward from different situations.
The distinction matters when someone says that an agent has learned. It may have improved its predictions about an existing rule without discovering a better rule. Evaluating a policy and improving a policy are separate tasks. The supplied governance category is retained, but the underlying concept is a reinforcement-learning method.

How it works

A simple tabular learner records the current state, chosen action, resulting state and received reward. For each observed state - action pair, outcome counts can produce estimated transition probabilities. Reward estimates describe the immediate consequences; stochastic rewards require suitable estimation rather than assuming every observation is identical.
Policy evaluation then connects each state's value to its immediate reward and the discounted values of possible successor states. Updating the model can therefore change values beyond the most recently visited state. The passive version continues following its specified policy. A control version also needs a way to improve action choices and obtain evidence about alternatives.
For a small finite model, calculations can use tables and repeated updates or equation solving. Larger problems require additional approximation and computational choices; the term alone does not specify a neural architecture.

Three practical examples

A training simulator estimates the score expected under an existing robot navigation policy. The transition table improves as the simulated robot encounters different obstacles. This measures that policy before anyone proposes a replacement.
A fictional equipment scheduler follows a fixed inspection rule. Recorded transitions between working, degraded and stopped conditions support estimates of future accumulated costs. A cost model must preserve the timing of inspection and repair outcomes.
A classroom game gives a point for each completed turn before an episode ends. Students estimate the probability of another turn and compute expected discounted points. Changing the estimated probability changes the value even when the action rule stays unchanged.

Think of it this way

Think of updating a route map from journeys, then recalculating expected travel consequences using that map. The map is learned from experience; the calculation uses the map. Following the same route while learning more about its reliability is different from selecting a new route.

Worked example

Consider one nonterminal state S and a terminal state T. A fixed policy selects the only available action at S. Every transition out of S earns one point, including the transition to T. Once T is reached, there are no further rewards, so its continuation value is zero. Set the discount factor to 0.9.
Across ten observed transitions from S, four return to S and six reach T. The estimated probabilities are therefore 0.4 and 0.6. The estimated value satisfies:
V(S) = 1 + 0.9 × [0.4 × V(S) + 0.6 × 0].
Rearranging gives 0.64 × V(S) = 1, hence V(S) = 1.5625 points.
Suppose ten additional observations contain six returns and four terminations. The combined counts are ten of each, making the return probability 0.5. Re-evaluation gives V(S) = 1 / (1 − 0.9 × 0.5), approximately 1.8182 points.
The estimated value rises because the revised model predicts more opportunities to earn points. The policy has not improved. Neither estimate establishes the true transition probability, and this deliberately simple reward design would be unsuitable if repeated loops were undesirable.

Illustrative case study

Fictional Cedar Simulations evaluates a delivery robot's established routing rule. Its first model pools all corridor observations, although congestion differs between quiet and busy periods. The resulting value estimates look precise but predict busy-period performance poorly.
The team examines the state description and separates conditions that affect subsequent movement. It then compares model predictions with fresh simulated journeys and records where observations remain sparse. It keeps the routing rule fixed during this evaluation, making the before-and-after result a comparison of estimates rather than a claim of better control.
Only after that investigation does it test alternative routes in a separate experiment. The lesson is that solving a model accurately cannot compensate for an inadequate description of the environment.

Common misconceptions and mistakes

Confusing passive learning with an absence of learning: the environment model and value estimates still change.
Calling policy evaluation policy optimization: a better estimate does not itself change the selected actions.
Treating observed frequencies as certain probabilities: limited or unrepresentative experience leaves uncertainty.
Assuming the reward expresses the real objective perfectly: rewarding continued activity can encourage an outcome the business dislikes.
Calling every adaptive controller the same algorithm: implementation assumptions and terminology must be checked in context.

Business use

Use the concept to separate questions in a proposed decision system: what has been observed, what model has been learned, what future outcomes are calculated, and who authorizes changes to actions? Ask for evidence that model predictions remain useful under relevant operating conditions. Computational accuracy and operational usefulness need separate checks.

Check your understanding

Why does the example's value rise?

Answer: The estimated chance of another rewarded turn increases.

Did its action rule change?

Answer: No.

Why can a precise value be misleading?

Answer: It may precisely evaluate an inaccurate model or an unsuitable reward.

Related terms

Learn next

Draw a two-state process, specify reward timing and terminal behavior, and estimate transition probabilities from counts. Calculate its fixed-policy value before considering any changes to actions.

Policy value

Sources

Russell and Norvig, AIMA fourth-edition algorithms, Figure 22.2, passive adaptive dynamic programming. https://aima.cs.berkeley.edu/algorithms.pdf AIMA pseudocode, third-edition passive ADP agent and fixed-policy evaluation. https://github.com/aimacode/aima-pseudocode/blob/master/md/Passive-ADP-Agent.md Alice Gao, University of Waterloo, Reinforcement Learning 1 lecture notes. https://cs.uwaterloo.ca/~a23gao/cs486686_f21/lecture_notes/Lecture_20_on_Reinforcement_Learning_1.pdf The numerical process, business questions and Cedar Simulations case are original teaching illustrations.

Previous: Adaptive differential pulse-code modulation Next: Adaptive filtering