Must the experiment be sequential?

No. A complete design can be chosen in advance, or choices can adapt as results arrive.

Does the most uncertain measurement always have the greatest value?

No. Measurement uncertainty may be noise unrelated to the target, and information value depends on the model and objective.

Does design optimization guarantee a useful result?

No. It optimizes an expectation; a particular realized outcome may be less informative than hoped.

AI Glossary / Statistics and probability / Advanced

Bayesian experimental design

Definition

Bayesian experimental design selects an experiment by maximizing expected utility under a probabilistic model of unknown quantities and possible observations. The design may specify measurement locations, experimental settings, sample allocation, or which question to ask next. Utility expresses the experiment's objective, such as improving a decision, reducing estimation error, or gaining information.

What it means

Before collecting data, a team can assess possible results, their probabilities under current beliefs, and their usefulness.

Bayesian design averages across that uncertainty instead of evaluating only one assumed parameter value or one hoped-for observation. A prior and observation model supply the probabilities; an explicit utility supplies the definition of useful.

Information gain is a common objective, but it is not the only one. Learning a great deal about a parameter that does not affect the business decision may be less valuable than learning a little about a decisive uncertainty. Costs, time, feasibility, and consequences can be built into the design problem.

How it works

Define the unknown quantity theta, candidate design d, and possible observation y. Specify a prior for theta and a likelihood for y conditional on theta and d. Together they determine the predictive distribution of outcomes under each design.

For every candidate design, consider how each possible outcome would update the posterior and contribute to the selected utility. Average over the appropriate joint or predictive distribution, then choose a feasible design with the greatest expected utility.

Expected information gain commonly uses the expected Kullback-Leibler divergence from posterior to prior. Under the model, this equals the mutual information between the unknown quantity and the prospective observation. Other criteria may minimize expected posterior variance or maximize the quality of a later operational decision.

In sequential design, actual observations update beliefs before the next design is chosen. Optimizing the immediate next step need not maximize a whole sequence's total value. Realistic problems often require simulation or approximation because posterior calculations, outcome averaging, and design optimization are computationally demanding. The numerical uncertainty of those approximations also needs attention.

Three practical examples

Engineering tests: choose operating conditions that are expected to distinguish competing explanations of a component's behavior.
Sensor placement: select measurement locations that reduce uncertainty about a target region, accounting for access and measurement costs.
Interactive AI: choose a clarification question expected to improve a later answer or decision, while including the user's effort in the objective.

Think of it this way

Think of a detective choosing the next question. Its value depends on the answers it might produce and how they would change the investigation. Likelihood and usefulness of all possible answers matter.

Worked example

Suppose an unknown system is equally likely to be in state A or state B. The prior entropy is one bit. Two candidate tests each report either A or B and have symmetric errors.

Test 1 reports the true state with probability 0.90. Because the prior is balanced, either report occurs with probability 0.50, and the posterior probability of the reported state is 0.90. The posterior entropy after either result is -0.90 log2(0.90) - 0.10 log2(0.10), approximately 0.4690 bits. Expected information gain is 1 - 0.4690 = 0.5310 bits.

Test 2 reports the true state with probability 0.70. Its posterior entropy is approximately 0.8813 bits, giving expected information gain of 0.1187 bits. If both tests have the same cost and the objective is information about the state, Test 1 is preferred.

This comparison is prospective: neither test result has been observed. If Test 1 were much more costly, slower, or infeasible, information gain alone would not settle the choice. A revised objective must express how those differences matter rather than subtracting monetary costs directly from bits without a justified conversion.

Illustrative case study

Fictional case: Fern Instruments has enough laboratory time for only a few additional calibration runs. Several model parameters remain uncertain, but only some strongly affect the device's intended operating range.

The team compares candidate settings using a utility focused on predictive accuracy in that range. It simulates possible results, updates the model for those simulated outcomes, and estimates the expected benefit of each design.

After the first real run, an unexpected response suggests the model is missing an effect. The team investigates this discrepancy before continuing the planned sequence. It recognizes that a design optimized under a wrong model can collect data that appear informative while overlooking the actual source of error.

Common misconceptions and mistakes

Do not confuse a design with the data observed afterward. Design variables are chosen before the associated outcome is known.

Do not assume Bayesian design always means maximizing information gain. Different objectives can produce different optimal experiments.

Do not optimize using an imagined favorable result without averaging over other possible outcomes. That substitutes wishful thinking for expected utility.

Do not assume an optimal design is universally best. It is optimal relative to the model, prior, utility, constraints, and computational solution. Misspecified noise, omitted mechanisms, or an overly concentrated prior can change the recommendation. Adaptive data collection also requires an analysis that respects how observations were selected.

Business use

The business advantage is disciplined allocation of scarce data-collection resources. A team asks which feasible observation would best advance a specified objective.

Useful project records include candidate designs, prior assumptions, utility definition, costs, simulation settings, and sensitivity analyses. Compare the proposed strategy with simpler designs and assess whether its computational overhead is justified. For consequential applications, model checks and a process for responding to unexpected observations are as important as the optimization itself.

Check your understanding

Why is Test 1 preferred in the example?

Answer: Under equal costs and the stated model, it has higher expected information gain about the state.

Can a utility focused on downstream decisions choose differently?

Answer: Yes. Information and decision value need not rank experiments identically.

Related terms

Learn next

Compare information-gain and decision-focused utilities on a small discrete problem. Then investigate sensitivity to prior assumptions and the difference between choosing one next experiment and planning an entire adaptive sequence.

Information gain

Sources

Antony Overstall, Properties of Fisher information gain for Bayesian design of experiments, introduction and Bayesian decision-theoretic formulation: https://arxiv.org/pdf/2003.07315 Catanach and Das, Metrics for Bayesian Optimal Experiment Design under Model Misspecification, author abstract on model discrepancy and design objectives: https://arxiv.org/abs/2304.07949

Previous: Bayesian active learning by disagreement Next: Bayesian filtering