What makes a selected vector informative?

Its inclusion is scored by its effect on the current approximation under the method's selection criterion, rather than by a universal property of the data point.

Does it use every point in the final predictor?

The sparse representation uses an active subset, although selection can consider a much larger training collection.

Is the result an exact Gaussian-process posterior?

Generally no. Sparsity and, for classification, likelihood approximation introduce approximation choices.

AI Glossary / Probabilistic machine learning / Advanced

Informative vector machine

Definition

The informative vector machine (IVM) is a sparse Gaussian process method that builds a limited active set of training examples using information-based selection. It aims to retain useful probabilistic predictions while controlling the computational cost of a full Gaussian process treatment.

What it means

The informative vectors are selected examples that support the model's approximation. They are not an embedding database, a vector-search engine or a special kind of computer hardware.

The original method combines an approximate Gaussian-process posterior with forward selection. Rather than searching all possible subsets of training data, it adds useful points progressively according to a specified score.

This places the method among sparse probabilistic learning approaches. Its active set is related in spirit to selecting a compact representation, but it is not obtained by the support vector machine's margin optimization. Similar-looking sparse predictors can have different objectives and uncertainty interpretations.

How it works

In the original classification construction, a Gaussian process prior is combined with a classification likelihood. The non-Gaussian posterior is approximated through Gaussian site factors associated with selected observations.

The active set begins small and grows one point at a time. A candidate's inclusion is assessed using an information-based criterion, such as the decrease in entropy of the current Gaussian approximation. The approximation is then updated before the next choice, so a point's usefulness depends on what has already been included.

A desired active-set size or resource limit controls sparsity. Greedy selection makes the search manageable but does not establish that the final subset is globally optimal. The approximation and selection rule both contribute to the resulting predictions.

Three practical examples

A probabilistic classifier uses a bounded active set to reduce the cost of kernel-based prediction.
A research team studies information-based data selection for a sparse Gaussian process approximation.
An analyst compares predictive probabilities and resource use as the permitted active-set size changes.

Think of it this way

Think of selecting a small briefing pack from a large archive. A document is useful when it adds something to the pack already chosen, not simply because it is impressive in isolation.

The analogy is limited because IVM uses mathematical posterior updates and information criteria. A compact pack is not automatically complete, and a compact active set does not guarantee accurate uncertainty everywhere.

Worked example

Comparing hypothetical entropy reductions

A univariate Gaussian with variance v has differential entropy one half of log(2 pi e v). Therefore reducing variance from v_before to v_after decreases entropy by one half of log(v_before/v_after).

For a simplified teaching comparison, candidate A changes a relevant Gaussian marginal variance from 4 to 1. Its entropy decrease is 0.5 log(4), approximately 0.6931 nats. Candidate B changes variance from 9 to 4, giving 0.5 log(9/4), approximately 0.4055 nats.

Although B has the larger absolute variance reduction, 5 versus 3, A has the larger entropy reduction because the relative reduction is greater. A would be preferred if these were the applicable selection scores and all other conditions were equal.

This calculation illustrates the information criterion, not a full IVM training step. The original algorithm derives candidate updates from the likelihood and current GP approximation; one cannot simply invent before-and-after variances and call them trained results. After a real inclusion, later scores must reflect the updated approximation.

Illustrative case study

Fictional case: limiting a classification model

A fictional research group needs a probabilistic classifier for a collection of technical measurements. It tests IVM with several active-set budgets and compares the results with a simpler baseline.

The smallest model predicts common cases reasonably but is less reliable in a sparsely represented region. The group examines whether a larger set, a revised kernel or another approximation better addresses that issue. It checks probability calibration as well as classification error.

The report records active-set sizes and measured runtime on its actual hardware. It does not quote historical benchmark speedups as though they were guaranteed for this project. The case illustrates evaluation choices without fabricating an experiment's outcome statistics.

Common misconceptions and mistakes

IVM is not another name for an SVM. The original IVM is a probabilistic Gaussian-process approximation with information-based selection, whereas standard SVM learning follows a different optimization formulation.

Its selection is also not automatically active learning. The original procedure can choose among already labeled training examples to build its approximation. Requesting new labels from an external source is a different data-acquisition problem, even though related information criteria can be used.

A predictive probability is not automatically calibrated merely because it comes from a probabilistic model. Approximation error, kernel mismatch and changed data distributions can all matter.

Business use

The method is relevant when a project wants a compact kernel-based probabilistic predictor and must manage computational resources. Define how many active points can be supported and which uncertainty properties matter for the intended decision.

Compare active-set sizes using separate evaluation data. Review the coverage of important input regions and the stability of selected examples. Measure prediction latency for the quantities actually requested: computing a mean or class decision can have different costs from computing a predictive variance. Keep historical research comparisons separate from evidence about the current implementation.

Check your understanding

A candidate changes a Gaussian variance from 8 to 2. Another changes it from 2 to 0.5. Under the stated scalar entropy-reduction calculation, which has the larger score?

Answer: They tie. Both variance ratios are 4, so each decrease is 0.5 log(4), approximately 0.6931 nats. Absolute variance reductions alone do not determine this score.

Related terms

Gaussian process
Differential entropy
Support vector machine
sparse Gaussian process
active set
assumed density filtering
predictive uncertainty

Learn next

Compare subset selection for a fixed labeled dataset with active learning that requests new labels. Then inspect how a GP's mean and uncertainty change when a new site enters its active set.

Active learning

Sources

Primary source: Lawrence, Seeger and Herbrich, Fast Sparse Gaussian Process Methods: The Informative Vector Machine, NeurIPS 2002: https://proceedings.neurips.cc/paper_files/paper/2002/file/d4dd111a4fd973394238aca5c05bebe3-Paper.pdf

The entropy comparisons and classification-project case are original illustrations. They do not reproduce the paper's benchmark results or full update algorithm.

Previous: Information value theory Next: InfoVAE