Can all training observations still be used?

Yes. They can all inform the reduced inducing representation.

Why is the covariance low rank?

Every latent function value is a linear combination of a finite vector of inducing values.

Does adding observation noise fix the approximation?

No. It models noisy measurements, not the omitted conditional function variation.

AI Glossary / Probabilistic machine learning / Advanced

Subset of regressors Gaussian process approximation

Definition

The subset of regressors Gaussian process approximation, or SoR, replaces a full Gaussian process covariance with a finite-rank covariance determined by a smaller set of inducing variables. It represents latent function values as linear combinations of those variables, discarding the residual conditional variation that the full Gaussian process would retain given them.

What it means

A full Gaussian process can represent variation beyond what is explained by a finite collection of function values. SoR restricts the function to the span of kernel-based features associated with selected inducing inputs. This makes calculations cheaper but changes the prior model.

Despite its name, SoR does not mean fitting an exact Gaussian process using only a subset of observed training records. All observations can contribute to estimating the smaller representation. The approximation is in the available function space and covariance. Inducing locations may be chosen from the training inputs or placed elsewhere, depending on the implementation.

How it works

Let U denote inducing locations and Kuu their kernel covariance. For input sets A and B, define Qab = Kau Kuu inverse Kub, assuming an invertible inducing covariance or a suitable numerical treatment. SoR uses Q in place of the full covariance K for training and test latent values.

Equivalently, with inducing values u, a zero-mean latent function is represented as f(x) = Kxu Kuu inverse u. Its conditional variance given u is set to zero. The observation-noise model remains separate, and noisy observations still update uncertainty about u.

The resulting latent covariance has rank at most the number of inducing variables. Matrix identities can reduce the cost when that number is much smaller than the training population. Accuracy depends on whether the inducing representation captures relevant variation. A major limitation is overly restrictive predictive uncertainty, particularly where covariance with all inducing locations becomes small.

Three practical examples

1. Scalable regression: use a limited kernel basis to fit many observations when a full covariance calculation is impractical. Compare both predictive means and uncertainty against a manageable exact reference.

2. Method benchmarking: use SoR as a simple baseline when evaluating richer inducing-point approximations. Keep the kernel and observation model comparable so the effect of residual covariance treatment is visible.

3. Compact surrogates: investigate a finite representation over a bounded operating region. Check extrapolation explicitly, because uncertainty outside that region can behave very differently from the full Gaussian process prior.

Think of it this way

Think of reconstructing a landscape using only a small set of fixed shapes. Their coefficients can remain uncertain, but the reconstruction cannot express a shape outside their span. If every chosen shape becomes nearly zero in a distant region, the reconstruction may become confidently flat there even though the real landscape is unknown.

Worked example

Consider an original one-inducing-point example. Use the squared exponential kernel k(x, z) = exp of minus (x minus z) squared divided by 2, with unit prior variance. Put the single inducing location at zero, so u = f(0) has prior variance 1.

Observe y = 2 at x = 0 with independent Gaussian noise variance 1. The posterior for u has mean 1 and variance 0.5. Under SoR, f(x) = k(x, 0)u exactly within the approximate model.

At x = 2, k(2, 0) = exp(minus 2), approximately 0.1353. The SoR latent predictive mean is therefore 0.1353, and its variance is 0.5 exp(minus 4), approximately 0.0092.

For the full Gaussian process with the same single observation, the predictive mean is also 0.1353, but the latent variance is 1 minus 0.5 exp(minus 4), approximately 0.9908. The mean agrees in this special example while uncertainty differs sharply. Far from zero, SoR's latent variance tends to zero, whereas the full model's returns to its prior variance 1.

These are latent-function variances. To predict a new noisy observation with variance 1, add that observation variance to each respective latent variance. Doing so does not restore the missing latent uncertainty in SoR.

Illustrative case study

Fictional case: Rowan Engineering builds a Gaussian process surrogate from laboratory measurements. Its analysts test SoR with a small inducing set and find that a satisfactory mean fit is insufficient evidence of a reliable uncertainty model. They deliberately evaluate points near the inducing locations and points near the limits of the operating domain.

The team compares the covariance treatment with other sparse approximations and checks uncertainty coverage on held-out experiments. If SoR is too confident in poorly represented regions, it changes the approximation rather than interpreting narrow intervals as strong evidence. This fictional case describes a validation strategy without claiming a measured result.

Common misconceptions and mistakes

SoR is not the same as selecting a subset of training data. It does not preserve the original full Gaussian process prior merely because it uses the same named kernel. Zero conditional variance given inducing values does not mean zero posterior uncertainty everywhere; uncertainty about u remains. Also distinguish SoR from approximations that retain or correct residual variances, such as deterministic training conditional or fully independent training conditional constructions.

Business use

The business value is a simpler and potentially cheaper probabilistic regressor. The risk is that a convenient finite representation can understate uncertainty where decisions most need caution. Teams should record inducing locations, their count, covariance conditioning, observation noise, and the region covered by validation. Forecast accuracy and uncertainty quality should be assessed separately; a good point predictor can still provide misleading intervals.

Check your understanding

Answer: 1.

What is the SoR latent variance at x = 2 in the example?

Answer: Approximately 0.0092.

Answer: 2.

Why can the full-model variance be much larger despite the same mean?

Answer: It retains residual function uncertainty beyond the inducing value.

Answer: 3.

What happens far from the single inducing point for this kernel?

Answer: SoR's latent mean and variance approach zero under the stated zero-mean model.

Related terms

Gaussian process
inducing variable
low-rank approximation
Nyström approximation
subset of data
deterministic training conditional
fully independent training conditional

Learn next

Derive the covariance Q from a finite inducing representation. Then compare SoR with the full Gaussian process on a one-point example and study how other sparse approximations treat the discarded residual covariance.

Gaussian process

Sources

Quiñonero-Candela and Rasmussen, A Unifying View of Sparse Approximate Gaussian Process Regression, explains SoR through its effective prior and inducing conditionals: https://www.jmlr.org/papers/volume6/quinonero-candela05a/quinonero-candela05a.pdf

The one-point calculation and Rowan Engineering case are original teaching examples; the case is fictional.

Independent review consulted Section 4, Equations 11–15, of the original sparse-GP comparison.

Previous: Subproblem Next: Subsetting