Can different optimality criteria disagree?

Yes, because they reward different information properties.

Is an approximate design directly executable?

Not always; fractional allocations must be converted into feasible run counts and reassessed.

Can the plan change during an experiment?

A prespecified sequential design can adapt, but its inferential and practical consequences must be considered.

AI Glossary / Statistics and probability / Intermediate

Optimal experimental design

Definition

Optimal experimental design is the planning of measurements, treatment assignments, or experimental settings to optimize a defined measure of information or decision value. The optimum is calculated relative to a model, feasible designs, and resource constraints. It can involve a complete plan chosen in advance or a sequential procedure that selects later measurements using information from earlier ones.

What it means

The aim is to learn efficiently from an experiment. Instead of treating every possible measurement as equally valuable, the design asks which observations best support a particular estimation, prediction, or decision goal.
Different goals can favor different plans. Estimating one contrast accurately may require a different allocation from estimating all parameters jointly. A claim of optimality is therefore incomplete unless it names the objective and the assumptions under which the plan was selected.

How it works

Define the scientific or operational question and the quantity to estimate. Specify the response model, measurement error, experimental units, and feasible choices. Include constraints such as total cost, allowable settings, minimum group sizes, and practical requirements for implementation.
Translate the objective into a criterion. D-optimality uses the determinant of an information matrix to emphasize joint parameter information. A-optimality emphasizes a sum of coefficient variances in a chosen parameterization. A contrast-focused criterion can minimize the variance of one specified linear combination of parameters.
Prediction-oriented designs target uncertainty over a region or at selected locations. Bayesian designs can average a utility or information criterion over a prior distribution, acknowledging uncertainty about parameter values before the experiment.
Distinguish exact designs, which allocate an integer number of runs, from approximate designs, which may initially assign fractional proportions. Rounding a theoretical allocation can change its properties, so check the implementable plan.
For nonlinear models, information often depends on unknown parameters. A locally optimal design uses specified parameter values; Bayesian or robust approaches account for uncertainty in other ways. Sequential designs update knowledge as measurements arrive and may choose the next setting accordingly.
Optimization must be followed by design review. Check estimability, randomization, blocking, replication, sensitivity to assumptions, and the ability to detect lack of fit. Computational search can miss a global optimum, and a mathematically optimal plan can still be impractical or fragile.

Three practical examples

Laboratory research: allocate a limited measurement budget to settings that best estimate a calibration parameter.
Product experiments: plan group sizes around a prespecified comparison and credible assumptions about outcome variation.
Scientific machine learning: select simulation runs that reduce uncertainty about a surrogate model or a decision-relevant quantity.

Think of it this way

Think of choosing questions for a short interview. If the goal is to understand one issue deeply, you allocate questions differently than if the goal is to cover several issues evenly. Efficiency is meaningful only after deciding what the interview should teach you.

Worked example

Consider an original planning problem comparing the means of two independent populations, A and B. Assume known variances of 4 and 16, equal observation costs, and a total budget of 12 observations. Let n observations go to A and 12 - n to B.
The variance of the difference between sample means is V(n) = 4/n + 16/(12 - n). Equal allocation gives V(6) = 4/6 + 16/6 = 10/3, approximately 3.333.
Differentiating the continuous allocation criterion gives -4/n^2 + 16/(12 - n)^2. Setting it to zero gives 12 - n = 2n, so n = 4. The resulting allocation is four observations in A and eight in B, with variance 4/4 + 16/8 = 3.
The second derivative is positive throughout the feasible interval, and checking all integer allocations confirms this minimum. Compared with equal allocation, the variance reduction is (10/3 - 3)/(10/3) = 10%.
A joint determinant criterion for estimating both means would instead maximize n(12 - n), giving equal allocation. This contrast shows why optimality depends on the goal. The tiny budget is illustrative, not a recommendation for an adequately powered real experiment.

Illustrative case study

Fictional case: Cedar Calibration has limited access to a reference instrument. Researchers initially plan equal measurements across two operating regimes but identify a single difference between regime means as their primary target.
Using justified variance estimates, they compare allocations and assess how sensitive the choice is to those estimates. They also retain enough observations for diagnostics and randomize execution. The final plan documents both its statistical objective and the practical constraints that prevented a purely mathematical allocation.

Common misconceptions and mistakes

An optimal experiment is not necessarily the experiment that produces the best-performing treatment during data collection. Exploration for information and exploitation of a promising setting can be different goals.
Do not assume the same design remains optimal when variances, costs, model terms, or target contrasts change. Estimated planning inputs can be uncertain, especially after a small pilot.
A sequential Bayesian design is not automatically appropriate for every application, and updating choices requires a compatible analysis and operational plan.
Finally, optimization does not replace measurement quality or scientific judgment. A design can precisely estimate a quantity that is poorly defined or irrelevant to the decision.

Business use

Businesses can use optimal experimental design when tests are costly, candidate settings are numerous, or information must arrive quickly. Planning should make the objective and assumptions reviewable before resources are committed. Compare expected gains with implementation complexity and sensitivity to model errors. Document why a chosen design departs from a standard balanced plan so later reviewers can reconstruct the reasoning.

Check your understanding

Why did the example allocate more observations to B?

Answer: Its larger variance made additional observations more valuable for estimating the difference.

Why did the joint determinant objective prefer equal allocation?

Answer: It optimized a different feature of the two-parameter uncertainty matrix.

Related terms

Optimal design
Bayesian experimental design
Contrast
Allocation ratio
Robustness
Fisher information
design criterion
sequential design

Learn next

compare a contrast-focused allocation with a joint-parameter design, then vary the planning variances and costs to assess sensitivity.

Contrast

Sources

Sources: NIST/SEMATECH, optimal-design criteria: https://www.itl.nist.gov/div898/handbook/pmd/section3/pmd34.htm and computer-aided design: https://www.itl.nist.gov/div898/handbook/pri/section5/pri52.htm . NIST OptBayesExpt manual: https://pages.nist.gov/optbayesexpt/manual.html . Primary methodological pages were opened. The allocation derivation, integer check, and fictional case are original.

Previous: Optimal discriminant analysis and classification tree analysis Next: Optimal matching