Is it always closed form?

No. Gaussian regression has conjugate formulas under the stated assumptions; classification generally needs additional inference.

Can the features be learned from the same training data?

Yes, but the resulting uncertainty remains conditional on those selected features.

Does unfamiliar input guarantee high uncertainty?

No. The representation may map it to familiar feature values.

AI Glossary / Probabilistic machine learning / Advanced

Neural-linear approximation

Definition

A neural-linear approximation combines a neural feature extractor with a Bayesian linear output model. The feature extractor is represented by fixed parameter estimates, while uncertainty is modeled over the final linear weights. It provides a limited form of Bayesian prediction without maintaining a posterior over every neural-network parameter.

What it means

A network first converts an input into useful features. A probability distribution over the output weights then describes several plausible ways to combine those features. Predictions average over these possibilities and can include uncertainty from the output layer.
The overall mapping can remain nonlinear in the original input. The word linear refers to the final combination of learned features. The approximation conditions on the chosen representation: uncertainty about how that representation should have been learned is not automatically included. Consequently, a narrow interval can still accompany a poor representation.

How it works

For regression, write y = phi(x) transpose w + noise. Here phi(x) is the fixed feature vector and w contains the uncertain output weights. With a Gaussian prior on w and Gaussian observation noise of known variance, Bayesian linear regression gives a Gaussian posterior analytically.
Prediction combines the posterior mean with the new feature vector. The variance of the latent prediction comes from uncertainty in w; predicting a new noisy observation adds the noise variance. These are different quantities and should be labeled separately.
One approach trains the neural network conventionally, freezes its features and replaces its output layer with Bayesian regression. Other training procedures are possible. If the representation later changes, previously stored feature-based statistics generally need reconsideration. Classification likelihoods do not usually retain the same closed-form Gaussian posterior.

Three practical examples

Experiment planning: a team uses neural features with a probabilistic regression head to represent an expensive experiment's response. It evaluates whether the resulting uncertainty is useful for choosing additional trials.
Small-data adaptation: an analyst freezes an existing encoder and learns a Bayesian output layer for a new numerical target. Validation determines whether the encoder retains the relevant information.
Sequential monitoring: a fixed representation supports updates to linear-model sufficient statistics as labeled observations arrive. A separate check watches for changes that make the representation unsuitable.

Think of it this way

Think of a fixed translation system followed by a panel of interpreters. The translator converts complex inputs into a compact vocabulary. The panel disagrees about how that vocabulary predicts an outcome. Neural-linear uncertainty represents the interpreters' disagreement, but does not automatically account for errors in the translation itself.

Worked example

Consider an original, stipulated two-feature regression example. The feature extractor is already fixed; we do not claim to have trained it here. Let the output weights have independent standard normal priors, and let observation noise have variance 1. There is no additional intercept.
The first observation has features (1, 0) and target 2. The second has features (0, 1) and target -1. The prior precision is the identity matrix. Adding the two feature outer products gives posterior precision 2 times the identity, so posterior covariance is 0.5 times the identity. Multiplying this covariance by the feature-weighted target sum (2, -1) gives posterior mean (1, -0.5).
For a new input represented by (1, 1), the predictive mean is 0.5. Its latent variance is 0.5 + 0.5 = 1. Its observation variance is 1 + 1 = 2. The second addition represents measurement noise, not extra uncertainty about the weights.
Now observe a third target of 1 at features (1, 1). The updated precision has rows (3, 1) and (1, 3). Its inverse has rows (3, -1) and (-1, 3), divided by 8. The updated target sum is (3, 0), giving weight mean (9/8, -3/8). At features (1, 1), the new predictive mean is 3/4, latent variance is 1/2 and observation variance is 3/2. This calculation illustrates exact inference conditional on the fixed features and stated Gaussian assumptions, not exact Bayesian inference for the complete neural network.

Illustrative case study

Fictional company Birch Testing predicts a laboratory measurement from sensor records. Its analysts train a feature extractor using the training partition, freeze it and fit a Gaussian Bayesian output layer. They keep a separate validation partition for assessing predictive errors and interval coverage.
The team notices that apparently confident predictions can still fail for a different sensor configuration. It investigates whether the encoder maps these unfamiliar records into feature regions already occupied by familiar records. It compares the approach with other uncertainty models before using it for trial selection. This is a fictional decision process; no improvement or deployment result is asserted.

Common misconceptions and mistakes

Bayesian output weights do not make every network parameter Bayesian. Nonlinear features do not make the final regression nonlinear in its weights. A posterior conditional on fixed features can be analytically exact while still forming an approximation to a richer model.
Do not confuse neural-linear inference with a Taylor expansion of the entire network around fitted parameters. Also avoid treating observation variance as purely reducible uncertainty: repeated relevant measurements can reduce weight uncertainty while the assumed noise component remains. Retraining an encoder without updating incompatible stored features is another practical mistake.

Business use

The approach offers a compromise between flexible representation and manageable probabilistic inference. It can be useful when a stable encoder is available and the final feature dimension is moderate. Teams should evaluate predictive calibration, representation drift and the cost of maintaining the encoder. A computational saving is valuable only when the resulting uncertainty supports the intended decisions.

Check your understanding

Why is the first observation variance 2?

Answer: Latent variance 1 and noise variance 1 are added.

What uncertainty is omitted?

Answer: Uncertainty over the fixed feature extractor's parameters.

Why revisit stored statistics after encoder retraining?

Answer: The same raw inputs may now produce different feature vectors.

Related terms

Learn next

Work through Gaussian linear regression with two features, then examine how changing those features affects both predictions and uncertainty. Compare a last-layer posterior with whole-network uncertainty approximations.

Linear regression

Sources

Snoek and colleagues, Scalable Bayesian Optimization Using Deep Neural Networks, 2015, Section 3, describes learned neural bases with Bayesian linear output weights: https://proceedings.mlr.press/v37/snoek15.pdf . The numerical calculation and fictional case are original illustrations. Section 3 equations 4–7 independently consulted for the posterior and predictive moments.

Previous: Neural gas Next: Neural machine translation