Are training function values independent?

Only after conditioning on the inducing variables in the approximation. Marginal dependence remains through those shared variables.

What does the diagonal correction preserve?

The original prior variance of each training function value, not every covariance or posterior quantity.

Must inducing inputs be actual training locations?

No. They may be selected from training inputs or treated as separate locations, depending on the method used.

AI Glossary / Probabilistic machine learning / Advanced

Fully independent training conditional approximation

Definition

The fully independent training conditional approximation (FITC) is a sparse Gaussian process regression approximation that makes training function values conditionally independent given a smaller set of inducing variables. It preserves each training value's conditional variance while dropping residual cross-covariances. This produces a low-rank covariance contribution plus a diagonal correction, reducing the cost of inference.

What it means

A Gaussian process describes correlated function values through a kernel. Standard dense regression becomes expensive as the number of training observations grows. Inducing variables summarize part of the function through values at selected or learned inducing locations. FITC uses them to retain a shared source of dependence while simplifying what remains.

Fully independent does not mean the training predictions are unrelated. They are independent only conditional on the inducing variables under the approximation. Once those shared variables are integrated out, training values remain correlated. The word training matters because the original FITC formulation treats training and test conditionals differently.

How it works

Let K_ff be the exact prior covariance of training function values, K_uu the covariance of inducing variables, and K_fu their cross-covariance. Define Q_ff = K_fu K_uu inverse K_uf. The exact training conditional has covariance K_ff − Q_ff. FITC replaces this conditional covariance by its diagonal.

After integrating the inducing variables, the approximate training covariance becomes Q_ff + diag(K_ff − Q_ff). Independent observation-noise variance is added separately. The diagonal correction restores original marginal training variances; the off-diagonal training covariances come from Q_ff.

With n training points and m inducing points, conventional dense algebra can exploit this structure at roughly O(n m squared + m cubed) cost rather than O(n cubed), with m much smaller than n. Implementation and storage strategies affect practical cost. Inducing locations and kernel parameters still need selection, and the approximation changes the model's covariance relationships.

Three practical examples

1. Sensor regression: model a large collection of calibration observations using fewer inducing locations. Check predictive uncertainty near operating conditions poorly represented by those locations.

2. Experimental modeling: build a probabilistic response surrogate when a full dense covariance factorization is inconvenient. Compare the approximation against a manageable exact subset before using uncertainty to choose experiments.

3. Spatial estimation: approximate a smooth response over many locations while retaining individual prior variances. Review whether the inducing representation captures local structure that matters for the intended predictions.

Think of it this way

Think of describing many local conditions through a small set of shared reference stations, while keeping a separate residual allowance for each location. The shared stations create dependence, but the remaining allowances no longer move together in the approximation. The analogy is mathematical: inducing points need not be physical sensors, and the residual correction is not automatically measurement noise.

Worked example

Consider two training function values with prior covariance K_ff having diagonal entries 1 and off-diagonal entries 0.7. Use one inducing variable with variance 1 and cross-covariances 0.8 and 0.5 with the training values. Then Q_ff is the outer product of (0.8, 0.5), giving diagonal entries 0.64 and 0.25 and off-diagonal entries 0.4.

The exact conditional residual covariance K_ff − Q_ff has diagonal entries 0.36 and 0.75 and off-diagonal entries 0.3. It is a valid positive-definite residual matrix: its determinant is 0.36 × 0.75 − 0.3 squared = 0.18.

FITC discards those residual off-diagonal entries and adds the retained diagonal to Q_ff. Its approximate prior covariance therefore has diagonal entries 1 and off-diagonal entries 0.4. The marginal variances are preserved, but the original covariance 0.7 is not.

If independent observation noise has variance 0.1, the observation covariance diagonal becomes 1.1 while the off-diagonal remains 0.4. This calculation demonstrates the approximation's structure; it does not compute a posterior prediction or establish an accuracy guarantee.

Illustrative case study

Fictional case: Juniper Controls fits a response model to a growing calibration archive. The analyst considers FITC to reduce computation and checks predictions against an exact Gaussian process on a smaller representative dataset. The team examines uncertainty as well as average errors.

Some operating regions are poorly covered by the inducing locations, prompting a review of their placement and number. The team also keeps the diagonal approximation correction separate from the instrument-noise interpretation. This fictional case illustrates approximation assessment, not a claim that FITC always preserves decision quality.

Common misconceptions and mistakes

FITC is not simply training an exact Gaussian process on a subset and discarding the remaining observations. All training observations can contribute through the approximation. Preserving covariance diagonal entries does not preserve the full covariance, posterior uncertainty or predictive accuracy.

The original training-only independence approximation should not be silently equated with applying independence to every test conditional. Extending the assumption to test values leads to the related fully independent conditional formulation, with different joint-prediction semantics. FITC is also distinct from variational inducing-point methods, even when both use similar matrix components.

Business use

Assess the tradeoff at the dataset size and prediction workload the business actually faces. Record inducing-point count, placement procedure, kernel, noise model and the predictive quantities used. A method that is adequate for pointwise means may need more scrutiny for joint uncertainty or downstream optimization.

Compare results as the inducing representation changes and test difficult regions, not only a random average. Inspect numerical conditioning and keep training and test conventions explicit. Approximation speed is useful only if the uncertainty and errors remain suitable for the decisions the model supports.

Check your understanding

Why does the worked approximation retain variance 1 but reduce covariance from 0.7 to 0.4?

Answer: It restores the residual diagonal while discarding the residual cross-covariance 0.3. Only the inducing-mediated covariance 0.4 remains between the two training values.

Related terms

Gaussian process
Covariance matrix
Conditional independence
inducing variable
sparse Gaussian process
low-rank approximation
fully independent conditional approximation
variational inference

Learn next

Derive the Gaussian conditional covariance for a small joint model, then compare keeping all, none or only diagonal residual covariance. Next examine how those alternatives change predictive variances and joint test predictions.

Sources

Quiñonero-Candela and Rasmussen, A Unifying View of Sparse Approximate Gaussian Process Regression, Section 6: https://jmlr.csail.mit.edu/papers/volume6/quinonero-candela05a/quinonero-candela05a.pdf

Independent review consulted section 6, equations 21–25 and the FIC distinction. Original covariance calculation verified; fictional case remains illustrative.

Previous: Fully connected layer Next: Function approximator