Does online planning mean internet access?

No. It means planning as decisions and observations arrive.

Does POMCP require the true current state?

It samples possible states from its belief rather than observing the true state directly.

Is every implementation's belief update identical?

No. Some implementations separate belief filtering from tree search.

AI Glossary / Graphs and symbolic AI / Advanced

POMCP

Definition

POMCP stands for Partially Observable Monte Carlo Planning. It is an online planning algorithm for partially observable Markov decision processes, combining particle-based belief representation with Monte Carlo tree search. It evaluates possible actions through simulated futures when the agent cannot directly observe the complete state of its environment.

What it means

An agent may know what it has done and what its sensors reported, yet remain uncertain about what is actually happening. A robot, for instance, can receive the same reading from several possible locations. Planning from one assumed location can overlook important alternatives. POMCP samples possible states and evaluates decisions across simulated action-and-observation histories. An informative action can be worthwhile because its observation changes what the agent should do afterward, even if obtaining that observation incurs a cost.

How it works

Each simulation starts from a state sampled from the current belief. A generative simulator produces a next state, observation and reward for a chosen action. The search tree distinguishes histories, rather than treating a sampled hidden state as information the real agent possesses. A UCT-style rule balances promising actions with less-explored alternatives. Rollouts estimate continuation value beyond the expanded tree, and sampled returns update action-value estimates.
The original algorithm also stores simulated states as particles at history nodes. After a real action and observation, the matching branch becomes the new root; its particles support the next belief. Search resumes from that updated situation. The implementation needs an initial-state sampling mechanism and a simulator; it does not require complete transition and observation tables.

Three practical examples

A robot in a warehouse compares moving toward a destination with taking another localization reading. It must account for possible positions and the information produced by its actions.
An equipment-support assistant plans diagnostic tests before choosing a repair. A test result can distinguish faults that initially look similar, changing the value of subsequent repair choices.
A simulated inspection workflow compares accepting, rejecting or examining a sealed item. Examination has a cost but can reveal information needed for the final disposition. Its usefulness depends on error rates, consequences and remaining opportunities to act.

Think of it this way

Imagine rehearsing a decision with several plausible versions of the situation hidden in envelopes. In each rehearsal, you receive only the clues your chosen actions would reveal. You compare the outcomes of decisions made from those clues. Opening every envelope before deciding would give the rehearsal an unfair advantage.

Worked example

Consider a fictional sealed crate that is good with probability 0.6 and defective with probability 0.4. There are three initial actions. Shipping ends the episode and gives reward 10 for a good crate or minus 10 for a defective one. Discarding ends the episode with reward 0. Inspecting costs 1, reveals the condition perfectly, and allows one final shipping-or-discarding decision. There is no discounting, and inspection does not change the crate's condition.
Shipping immediately has expected return 0.6 times 10 plus 0.4 times -10 = 2. Discarding returns 0. Inspecting, then shipping only good crates, returns 9 when the crate is good and minus 1 otherwise. Its expected return is 0.6 times 9 plus 0.4 times -1 = 5. Inspection is the best of these options.
For an illustrative set of ten simulations containing six good and four defective starting states, the shipment returns sum to 20 and average 2. The inspection-policy returns sum to 50 and average 5. These deliberately specified samples reproduce the exact expectations; actual random samples need not contain that proportion.
After a perfect 'good' inspection result, the posterior probability of good is 1 under this model. The final decision can then use that observation. A planner that secretly reads the hidden condition before its first action would ship good crates and discard defective ones for an apparent value of 6 without paying inspection cost. That is unavailable information, not a better feasible policy. This calculation illustrates the planning problem; it is not a reported execution trace of POMCP.

Illustrative case study

In a fictional distribution center, the crate example becomes a simulator test. The team checks that the planner's decision interface receives observations, while the simulator separately maintains the hidden condition. A test fails if the planner obtains the cost-free value of 6 through accidental access to that condition.
The team then replaces perfect inspection with measured error rates and compares policies across repeated runs. It records inspection frequency, incorrect shipments and total return. It also varies the simulation budget. The original values provide a clear baseline for checking the model, while the revised experiment addresses a different, noisier decision problem.

Common misconceptions and mistakes

Particles are possible states, not additional observations from the real world. More simulations cannot correct a simulator that describes the wrong dynamics. A small particle collection can omit a possibility that matters, and a finite search budget does not guarantee the best action.
Implementation names also need care. BasicPOMCP.jl implements the PO-UCT search component but uses a separate belief updater rather than the original reuse of tree particles. That distinction matters when interpreting an implementation's behavior. The original paper's convergence argument is conditional; it does not certify every finite run with an approximate belief.

Business use

POMCP can support experiments involving sequential decisions, hidden conditions and informative observations. Start with a testable simulator and specify what the decision maker may know at each step. Compare against a simple policy and an exact solution on small cases when possible. Evaluate errors in both belief estimation and action selection. In the crate example, changing inspection cost from 1 to 5 reduces its expected return from 5 to 1, making immediate shipment's return of 2 preferable. The objective determines which information is worth buying.

Check your understanding

Why is the crate's cost-free value of 6 misleading?

Answer: It assumes condition information unavailable before inspection.

What is inspection's expected return at cost 1?

Answer: Five.

Can ten simulated crates establish exact performance in a realistic deployment?

Answer: No; the specified ten-sample illustration only reproduces this simple model's arithmetic.

Related terms

Belief state
Monte Carlo tree search
UCT
Generative model
POMDP
particle filter
rollout
value of information
online planning

Learn next

Rework the crate example with an imperfect inspection test. Calculate the posterior after each result before deciding whether shipment remains worthwhile.

Sources

Silver and Veness, Monte-Carlo Planning in Large POMDPs: https://papers.nips.cc/paper_files/paper/2010/file/edfbe1afcf9246bb0d40eb4d8027d90f-Paper.pdf JuliaPOMDP, BasicPOMCP implementation and belief-update distinctions: https://github.com/JuliaPOMDP/BasicPOMCP.jl AIMA glossary, POMCP: https://github.com/aimacode/aima-glossary/blob/master/glossary.md

Previous: Polysemanticity Next: Pooled variance