Is every kernel a covariance function?

No; validity requires the relevant symmetry and positive-semidefinite properties.

Does nearby always mean correlated?

Only according to the chosen input representation and covariance rule.

Can covariance vary by location?

Yes. Nonstationary functions can represent location-dependent covariance, while stationary functions impose translation-invariant structure.

AI Glossary / Statistics and probability / Advanced

Covariance function

Definition

A covariance function specifies the covariance between a random process's values at two inputs. Written k(x,x'), it describes how their deviations from the mean vary together. In Gaussian-process modeling, it encodes assumptions about patterns such as smoothness, scale, and periodicity, and generates covariance matrices for selected input points.

What it means

A random process assigns a random quantity to each input, such as a time, location, or machine setting. Rather than specifying a separate covariance for every conceivable pair, a covariance function supplies a consistent rule.

For a real-valued process f with mean m, k(x,x') = E[(f(x) - m(x))(f(x') - m(x'))]. This is covariance of uncertain function values, not simply covariance of the input coordinates themselves.

The rule is often called a kernel in machine learning. A general similarity function is not automatically a valid covariance function. Its pairwise values must fit together into a mathematically possible collection of variances and covariances.

How it works

Choose a covariance family appropriate to the modeling question and input representation. For example, the squared-exponential form is k(x,x') = sigma_f squared × exp[-(x - x') squared/(2ell squared)] in one dimension. Its amplitude sets marginal variance and its length scale controls how quickly covariance falls with distance.

For any finite set of inputs, form K with entries K_ij = k(x_i,x_j). A real covariance function must produce symmetric positive-semidefinite matrices: every linear combination of function values must have nonnegative variance.

The diagonal k(x,x) gives marginal variance. Off-diagonal values express co-movement. Independent observation noise is usually modeled separately by adding its variance to the observation covariance matrix's diagonal. Fitting parameters or choosing a family changes the assumptions; it does not guarantee that those assumptions describe the application well.

Three practical examples

A Gaussian-process model connects nearby machine settings when predicting product quality from a limited number of experiments.
A spatial model describes how measurement uncertainty at one site relates to uncertainty at another, using an appropriate distance or structure.
A demand model uses a periodic covariance component to express recurring patterns, while other components represent slower changes and irregular variation.

Think of it this way

Think of a rule describing how strongly two points on a flexible sheet tend to move together. Nearby points may move similarly, but the rule must remain physically coherent across the whole sheet. Assigning plausible-looking pairwise movements independently can produce an impossible combined structure.

Worked example

Consider an original prior covariance example using k(x,x') = 4 exp[-(x - x') squared/2], so the marginal variance is 4 and the length scale is 1. At inputs 0 and 1, k(0,0) = k(1,1) = 4 and k(0,1) = 4 exp(-1/2), approximately 2.4261.

The two-point covariance matrix is therefore [[4, 2.4261], [2.4261, 4]]. Its correlation is 2.4261/4, approximately 0.6065. Its eigenvalues are 4 + 2.4261 and 4 - 2.4261, both positive.

At inputs 0 and 2, covariance is 4 exp(-2), approximately 0.5413. The greater separation reduces covariance under this particular kernel.

Suppose measurements at 0 and 1 have independent noise of variance 1. Their observation covariance matrix becomes [[5, 2.4261], [2.4261, 5]]. Cross-covariance stays unchanged, while observation correlation becomes approximately 0.4852.

These are prior assumptions, not posterior predictions after observing data. The noise changes uncertainty about observations, not the latent function's stated prior covariance. Different kernel families can behave differently with distance.

Illustrative case study

Fictional case: Meridian Labs uses a Gaussian process to explore production settings. Its first kernel treats a small change in temperature and a small change in pressure as equivalent because both inputs are passed in raw numerical units.

The analysts inspect scaling and fit meaningful length-scale parameters. They then compare predictive behavior on held-out experiments, including settings near the boundaries of the observed range.

A smooth covariance model works adequately in one operating region but misses an abrupt process change elsewhere. Meridian records that limitation and revises its modeling strategy instead of interpreting narrow model uncertainty as proof that predictions are reliable. The example illustrates model checking, not a guaranteed benefit of a particular kernel.

Common misconceptions and mistakes

A covariance function is not the same object as the finite covariance matrix obtained by evaluating it.
Positive semidefinite does not mean every entry must be positive. Valid covariance functions can express negative covariance.
A stationary covariance depends on input differences; that alone does not establish strict stationarity of every possible process using it.
Length scale and observation-noise variance play different roles and can be difficult to distinguish with limited data.
A kernel's assumptions can be inappropriate even when its matrices are mathematically valid.
Numerical stabilization is not a substitute for examining model fit and meaningful measurement noise.

Business use

Covariance functions matter when AI systems learn from expensive experiments, interpolate measurements, or estimate uncertainty across related inputs. Teams should document what the selected family assumes about distance, smoothness, periodicity, and input scaling.

Assess predictions under realistic validation splits. A random split among nearby points may look strong while hiding poor performance at new locations or future times. The covariance structure should match the intended generalization task.

Also distinguish uncertainty conditional on fitted kernel parameters from uncertainty that accounts for parameter estimation. A convenient model output can understate broader uncertainty when the family or parameters are poorly determined. Decisions should consider those modeling limits alongside the numerical predictions.

Check your understanding

Answer: 1.

What does k(x,x) represent?

Answer: The marginal variance of the latent function value at x. 2.

Why add independent measurement noise only to the diagonal?

Answer: Noise in distinct observations has zero covariance under that assumption. 3.

Does a positive-definite matrix prove a useful model?

Answer: No; mathematical consistency and empirical suitability are different requirements.

Related terms

Gaussian process
Kernel
Covariance matrix
Stationarity
length scale
positive-semidefinite matrix
observation noise

Learn next

Learn covariance matrices and quadratic forms, then compare squared-exponential, Matérn, periodic, and linear covariance families. Practice constructing small matrices and separating latent-function uncertainty from measurement noise before studying Gaussian-process conditioning.

Gaussian process

Sources

Rasmussen and Williams, Gaussian Processes for Machine Learning, Chapter 4, Covariance Functions: https://gaussianprocess.org/gpml/chapters/RW4.pdf

The primary author chapter's definitions and validity conditions were read. The two-point calculation and fictional process-modeling case are original teaching examples; the worked matrix is a prior covariance illustration.

Previous: Covariance Next: Covariance graph