Is a Gaussian process required?

No. It is a common choice, but other probabilistic surrogates can support suitable acquisition methods.

Does Bayesian optimization optimize the surrogate mean?

Not generally. It selects evaluations by optimizing an acquisition function derived from the model and decision objective.

Can it handle noisy observations?

Yes, with appropriate models and acquisition methods; noiseless formulas should not be transferred without checking their assumptions.

AI Glossary / Statistics and probability / Intermediate

Bayesian optimization

Definition

Bayesian optimization is a method for choosing successive evaluations of an expensive objective using a probabilistic surrogate model and an acquisition rule. The surrogate represents uncertainty about the objective, while the acquisition rule balances opportunities to improve the result against opportunities to learn. It is particularly useful when evaluating a candidate costs much more than choosing one.

What it means

Suppose training one machine-learning configuration takes several hours. Evaluating every possible setting is impractical, but the results already collected can guide the next experiment.

Bayesian optimization builds a model of the relationship between settings and outcomes, then uses that model to decide where another evaluation is most valuable. Its uncertainty matters alongside its predicted average performance.

The term does not mean applying Bayes' theorem to guarantee the global best answer. It describes a sequential decision process whose effectiveness depends on the surrogate, the acquisition rule, the search space, and the available evaluation budget. Gaussian processes are common surrogates, but they are not the only possibility.

How it works

Define the objective, permitted inputs, measurement procedure, and evaluation budget. Collect initial observations that provide a useful starting picture of the search space. Fit a probabilistic surrogate to those observations, accounting for measurement noise when relevant.

Construct an acquisition function that scores candidate evaluations. Expected improvement, for example, measures the posterior expected amount by which a candidate exceeds an incumbent under its stated assumptions. Other acquisitions pursue different exploration, information, or decision objectives.

Optimize the acquisition function to select the next candidate, evaluate the expensive objective there, and update the surrogate with the new observation. Continue until the budget or an appropriate stopping rule is reached. Batch methods can choose several evaluations while accounting for their combined information rather than treating them as independent single selections.

The acquisition is itself an optimization problem and can have multiple local optima. Constraints, categorical settings, unreliable measurements, and high dimensionality require suitable modeling and search methods. Ordinary improvement formulas using a known best value need adaptation when the observed best result is noisy.

Three practical examples

Model development: select learning rates and regularization strengths when each training run is expensive.
Engineering experiments: choose permitted process settings to investigate while respecting operating constraints and measurement uncertainty.
Simulation studies: allocate a limited number of detailed simulator runs to promising or informative design alternatives.

Think of it this way

Think of exploring an unfamiliar landscape with costly surveying equipment. A rough map estimates both the terrain and how uncertain each region is. The next survey location can be attractive because it looks high or because its uncertainty leaves a meaningful chance of discovering something better.

Worked example

Consider maximizing an objective whose best evaluated value is exactly 10. Assume noiseless evaluations and normal posterior distributions for the latent value at two candidates. Use classical expected improvement with no additional improvement margin.

Candidate A has posterior mean 10.5 and standard deviation 0.2. Candidate B has mean 9.5 and standard deviation 2.0. For mean m, standard deviation s, and incumbent b, expected improvement is (m - b)Φ(z) + sφ(z), where z = (m - b)/s and Φ and φ are the standard normal distribution and density functions.

For A, z = 2.5 and expected improvement is approximately 0.500401. For B, z = -0.25 and expected improvement is approximately 0.572689. This acquisition therefore selects B, even though B has the lower predicted mean.

The larger uncertainty creates enough potential upside to outweigh that lower mean under this particular criterion. These values are expected gains in objective units, not probabilities of improvement. Neither value guarantees what the actual evaluation will return. A different acquisition, evaluation cost, or constraint could produce a different choice.

Illustrative case study

Fictional case study: Cedar Forecasting has a fixed budget for expensive training runs. Its team defines a validation score, locks the data split, and specifies feasible parameter ranges before beginning a Bayesian optimization pilot.

The team records every proposed setting, realized score, runtime, and failed run. It reviews unusually optimistic surrogate predictions and compares the completed search with a prespecified baseline using the same budget. A final untouched test set is used only after model selection.

The pilot illustrates disciplined experiment allocation. It does not establish that Bayesian optimization wins for every search space, nor that repeated validation-set optimization leaves generalization performance unbiased.

Common misconceptions and mistakes

Do not choose the highest surrogate mean and assume that this reproduces every Bayesian optimization method. The acquisition determines the selection policy.

Do not treat surrogate uncertainty as automatically calibrated or interpret a finite search as proof of global optimality. Poor input representations, misspecified noise, and changing objectives can undermine the process.

Avoid comparing methods using different evaluation budgets or allowing the final test set to guide repeated search decisions. Also distinguish failed evaluations from genuinely poor measured outcomes; failure handling can change the search substantially.

Business use

Businesses can use this approach to allocate costly experimentation more deliberately. The relevant comparison includes evaluation quality, total compute or experiment cost, decision time, and the consequences of selecting a disappointing candidate.

Before deployment, establish who approves feasible settings, how failures are recorded, and which evidence ends the search. Cheap objectives or very large unstructured search spaces may favor simpler alternatives. Operational benefits must be demonstrated on the actual task rather than inferred from the method's name.

Check your understanding

Why does candidate B win the example?

Answer: Its greater uncertainty produces higher expected improvement despite a lower mean.

Is 0.572689 a probability?

Answer: No. It is an expected improvement measured in the objective's units.

What should remain untouched during hyperparameter search?

Answer: The final test data reserved for evaluating the selected procedure.

Related terms

Surrogate model
Gaussian process
Expected improvement
acquisition function
hyperparameter optimization
exploration and exploitation
experimental design

Learn next

compare expected improvement with confidence-bound acquisitions, investigate noisy and constrained optimization, and study how batch selection accounts for pending evaluations. Practice checking the surrogate's predictive behavior before trusting its uncertainty-driven recommendations.

Sources

Sources: BoTorch, Bayesian optimization overview, https://botorch.org/docs/overview ; BayesianOptimization documentation, ExpectedImprovement, https://bayesian-optimization.github.io/BayesianOptimization/2.0.3/reference/acquisition/ExpectedImprovement.html ; Google Machine Learning Glossary, Bayesian optimization, https://developers.google.com/machine-learning/glossary#bayesian-optimization . Primary documentation was read. The numerical comparison is an original independently calculated illustration of classical noiseless expected improvement.

Previous: Bayesian neural network Next: Bayesian optimization acquisition function