Is policy search always neural?

No. Thresholds, linear functions and other representations can define the search space.

Does it need a simulator?

No. Policies can be evaluated through collected interaction data, although methods differ in which data they can validly use.

Is an actor-critic method policy search?

Its actor performs policy optimization while its critic supplies value estimates that support learning.

AI Glossary / Graphs and symbolic AI / Advanced

Policy search

Definition

Policy search is a reinforcement-learning approach that improves a decision rule by searching over candidate policies or their parameters. A policy specifies how an agent selects actions from available information. The objective usually measures expected cumulative reward, so the search concerns the consequences of repeated decisions rather than the attractiveness of one isolated action.

What it means

A warehouse controller might decide whether to keep a machine working or service it. Policy search adjusts the rule governing that choice, then evaluates the resulting sequences of operation. A policy can be a short threshold rule, a probability distribution over actions, or a neural network. Searching directly for policy parameters differs from first learning an action-value function through Bellman-based updates and extracting actions from it. Nevertheless, policy-search algorithms can also learn value functions to support their updates.

How it works

First specify the observations, permitted actions, reward, episode length or discounting, and policy family. Evaluate an initial policy using interaction data or an environment model. Then propose parameter changes, estimate their effect on return, and repeat within a defined evaluation budget.
Some methods estimate policy gradients; others explore parameter changes without computing a gradient. Exploration can occur through sampled actions or perturbed policy parameters. Actor-critic methods combine a policy, the actor, with value estimates from a critic. Model-based policy search evaluates or improves policies using an environment model; model-free methods work without that transition model. Noisy estimates, model errors and restricted policy families can all affect which policy appears best.

Three practical examples

A simulated delivery vehicle learns steering decisions from its position and speed. The return rewards route completion and penalizes specified violations. Simulator performance must be checked against the conditions of actual operation.
A service platform tests rules for assigning incoming jobs to available workers. The reward balances waiting time and operating cost, while action constraints exclude assignments that are impossible.
A game agent adjusts the probabilities of legal moves. It evaluates complete or truncated games rather than treating every immediate score increase as evidence of a better long-term policy.

Think of it this way

Imagine testing different instructions for a shift supervisor. Each instruction set explains what to do when particular situations arise. Policy search revises those instructions and compares the shifts they produce. It does not merely select the next task on today's list.

Worked example

Consider a fictional machine with two states: wear 0, meaning good, and wear 1, meaning service is due. It starts at 0. Each episode lasts exactly three decisions, with no discounting.
Producing in state 0 gives reward 5 and changes wear to 1. Producing in state 1 gives reward 1 and leaves wear at 1. Servicing in either state gives reward minus 2 and resets wear to 0.
Search three threshold policies: service when wear is at least theta; otherwise produce. The candidates are theta = 0, 1 and 2.
For theta = 0, the machine services at every decision. Its return is -2 - 2 - 2 = -6.
For theta = 1, it produces, services, then produces. Its return is 5 - 2 + 5 = 8.
For theta = 2, neither possible state reaches the threshold, so it produces three times. Its return is 5 + 1 + 1 = 7.
The search selects theta = 1 among these candidates. This is an exact comparison in a specified deterministic model, not an estimate from noisy trials. If servicing instead costs 4, that candidate returns 6 while theta = 2 still returns 7. Changing the reward changes the preferred policy. The result establishes the best tested threshold, without establishing optimality over every possible history-dependent rule.

Illustrative case study

In a fictional packaging business, an analyst uses this small example to explain a proposed maintenance experiment. A manager initially prefers always producing because servicing incurs an immediate cost. The three-decision comparison shows how a temporary loss can restore later productivity.
Before extending the experiment, the team adds realistic breakdown outcomes, operating limits and longer episodes to its simulator. It compares candidate policies across the same initial conditions and repeated trials. It also records uncertainty and checks whether policies exploit unrealistic simulator behavior. The example motivates a structured investigation; it does not demonstrate a maintenance saving for a real factory.

Common misconceptions and mistakes

A policy is not necessarily a fixed action sequence: it can select different actions when observations differ. Direct policy optimization does not prohibit value functions; a critic can improve learning. Nor does every accepted update guarantee a real improvement: estimated returns and gradients can be noisy.
Finding the best candidate in a restricted family does not prove global optimality. Average reward also does not automatically enforce a hard operating limit. Finally, a policy that exploits errors in a learned simulator may perform poorly in the environment the simulator represents.

Business use

Policy search suits problems where decisions influence later opportunities and outcomes. A useful business specification states the return, feasible actions, evaluation horizon and acceptable uncertainty. Compare against established rules, test across relevant starting conditions, and examine undesirable behavior alongside aggregate reward. Computational cost includes collecting interactions and evaluating proposals, not just updating parameters. A transparent threshold policy can provide a useful baseline before a more complicated policy is justified.

Check your understanding

Why does theta = 2 never service?

Answer: The available wear values are only 0 and 1.

Why can servicing help?

Answer: It changes the next state, allowing a later production reward of 5.

Does return 8 guarantee the policy will succeed on a real machine?

Answer: No. That value follows only from the stated model and objective.

Related terms

Learn next

Recalculate the threshold comparison for a two-decision episode. Then explain why a finite-horizon policy might include remaining time in its input rather than using wear alone.

Sources

Deisenroth, Neumann and Peters, A Survey on Policy Search for Robotics: https://www.ias.informatik.tu-darmstadt.de/uploads/Site/EditPublication/PolicySearchReview.pdf OpenAI, Spinning Up, kinds of reinforcement-learning algorithms: https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html AIMA glossary, policy search: https://github.com/aimacode/aima-glossary/blob/master/glossary.md

Previous: Policy loss Next: Policy value