Must the relationship be linear?

No. Kernel choice can support nonlinear function behavior.

Does the model always pass through training observations?

No. A noise model generally permits deviations.

Is the returned uncertainty complete?

No. It depends on the modeled sources of uncertainty and may condition on fitted parameters.

AI Glossary / Statistics and probability / Advanced

Gaussian Process Regression

Definition

Gaussian process regression is a probabilistic regression method that places a Gaussian process model over an unknown function and updates that model using observed data. A mean function and covariance kernel specify its prior behavior. With Gaussian observation noise, conditioning gives analytical expressions for the posterior mean and covariance at chosen inputs, conditional on the model parameters. The result includes predictions and model-based uncertainty. The method can learn nonlinear relationships even though its basic conditioning calculations use Gaussian distributions.

What it means

Instead of choosing one fixed curve at the outset, the model describes a distribution over possible functions. A kernel expresses which input settings should have related function values and how that relationship changes with distance or other structure. Observations make some functions more plausible and others less plausible. Gaussian refers to the joint distribution of function values at finitely many inputs; it does not require the inputs themselves to follow a Gaussian distribution. A predictive interval is meaningful relative to the chosen model and its assumptions, rather than a guarantee against every future error.

How it works

Specify y = f(x) + noise, a prior mean and kernel, and a noise covariance. For independent Gaussian noise with variance σ², let K be the training covariance, k* the vector of covariances between training points and a new point, and k** the new point's prior variance. With zero prior mean, the latent predictive mean is k*ᵀ(K + σ²I) inverse y and its variance is k** − k*ᵀ(K + σ²I) inverse k*. Implementations usually solve linear systems rather than explicitly invert matrices. Prediction of a new noisy observation adds its noise variance. Kernel parameters may be fitted or integrated, with different uncertainty implications.

Three practical examples

Process monitoring: estimate a smooth response across operating settings while highlighting regions supported by little data.
Scientific measurement: model a nonlinear calibration curve and distinguish uncertainty about the underlying response from measurement noise.
Experiment selection: use a GP's predictions and uncertainty to help choose the next evaluation in Bayesian optimization. The acquisition rule and experimental constraints remain separate parts of that workflow.

Think of it this way

Think of a collection of flexible curves connected by assumptions about smoothness and similarity. Observations pull the collection toward plausible responses. Where observations are noisy, the curves need not pass through every measured point. Where the kernel says a new location is only weakly connected to observed locations, the data have less influence. The flexibility comes from the covariance design, not from assuming that any curve can be learned reliably from a few points.

Worked example

Take one training observation y = 4. Assume zero prior means, latent training variance 2, latent test variance 2, covariance 1 between them, and independent observation-noise variance 1. The observed training variance is therefore 3. The posterior test mean is (1/3) × 4 = 4/3, approximately 1.333. Its latent variance is 2 − 1²/3 = 5/3, approximately 1.667. A new noisy measurement at that test input has variance 5/3 + 1 = 8/3, approximately 2.667. At the original training input, the latent posterior mean is (2/3) × 4 = 8/3 and variance 2 − 2²/3 = 2/3. It does not interpolate y = 4 because the observation is noisy. All results condition on the specified covariance and noise parameters; they do not include uncertainty from estimating those parameters.

Illustrative case study

Fictional case study: Willow Instruments fits a GP to calibration measurements taken at selected temperatures. An initial report shows narrow intervals for the latent response, which a manager interprets as intervals for the next individual measurement. The analyst corrects the report by adding the modeled measurement variance for that prediction target. The team also reserves complete temperature settings for validation rather than randomly splitting repeated measurements at identical settings. This exposes weaknesses in extrapolation and makes the displayed uncertainty better aligned with the decisions being considered.

Common misconceptions and mistakes

Common mistakes include using a kernel without checking whether its smoothness or periodicity matches the task, confusing latent-function intervals with observation intervals, and treating fitted hyperparameters as known without acknowledging the approximation. A small numerical stabilizer is not automatically a defensible measurement-noise model. Correlated observations require appropriate covariance treatment. Dense exact GP training typically takes cubic time and quadratic storage in the observation count, so larger datasets may need approximations or special structure. Good performance inside the observed region does not establish reliable extrapolation or causality.

Business use

For businesses, GP regression can be useful when observations are costly and uncertainty helps guide further measurement or review. Compare it with simpler baselines and assess both error and calibration on representative validation data. Document scaling, kernel choice, parameter fitting and the exact prediction target. A model suitable for selecting experiments may need different evaluation from one used to publish customer forecasts. Establish how the model will handle new input regions, changing noise and repeated observations before relying on its intervals operationally.

Check your understanding

Why is the test observation variance 8/3 while the latent variance is 5/3?

Answer: The new measurement includes independent noise with variance 1 in addition to uncertainty about the latent function. Adding variances is justified here by the specified independence assumption.

Related terms

Learn next

reproduce a small GP calculation by solving its covariance system, then compare noiseless and noisy predictions. Examine how kernel length scale changes interpolation and uncertainty, and assess predictive interval coverage on held-out inputs rather than judging only the fitted curve.

Sources

Sources: Rasmussen and Williams, Gaussian Processes for Machine Learning, chapter 2, https://gaussianprocess.org/gpml/chapters/RW2.pdf. Scikit-learn, Gaussian Processes, https://scikit-learn.org/stable/modules/gaussian_process.html. These primary references describe GP regression and implementation assumptions. The numerical calculation and fictional case are original teaching examples.

Previous: Gaussian process emulator Next: Gaussian random variable