Does the base kernel have to be an RBF kernel?

No. Other valid kernels can be used, depending on the modeling objective.

Is the whole model still a Gaussian process?

Conditional on the fixed feature-map and kernel parameters, the composed covariance defines a Gaussian process over the original inputs.

Does joint training remove overfitting?

No. A probabilistic objective does not eliminate the need for held-out evaluation and appropriate control of model complexity.

AI Glossary / Probabilistic machine learning / Intermediate

Deep kernel learning

Definition

Deep kernel learning combines a learned deep neural representation with a kernel method. In the Gaussian process formulation, a neural network transforms the inputs and a base covariance kernel compares the transformed values. The representation and kernel parameters can be learned jointly from the prediction task.

What it means

A kernel expresses which inputs should have related predictions. A simple kernel applied directly to raw inputs may regard two observations as dissimilar even when their task-relevant features are alike. A learned transformation changes that comparison.
The Gaussian process then operates on the learned features, providing a probabilistic output model. This does not mean every part of the system is treated probabilistically. When network weights are fitted as single values, the resulting uncertainty is conditional on those fitted choices; it does not automatically include uncertainty about all possible feature networks.

How it works

Write the transformed kernel as k_deep(x, x′) = k_base(g_w(x), g_w(x′)). Here g_w is the neural feature map, and k_base is a valid covariance kernel. For fixed network weights, applying a valid kernel to transformed inputs preserves its positive-semidefinite property.
In the original Gaussian process approach, learning adjusts network weights and kernel hyperparameters through the Gaussian process marginal likelihood. Predictions use the resulting covariance relationships and the observation model.
Practical implementations may use inducing variables, interpolation, or other approximations to control computation. Their speed and accuracy depend on the actual method and data. A scalability claim for one particular implementation should not be treated as a universal property of every deep kernel model.

Three practical examples

Equipment monitoring: learn representations of sensor windows, then compare windows through a kernel when predicting a measured operating quantity.
Image regression: use an image feature network before a Gaussian process to estimate a continuous target, checking whether the probabilistic predictions remain calibrated.
Scientific modeling: compare a learned feature-based kernel with a conventional kernel to determine whether the added flexibility captures meaningful nonlinear structure rather than fitting noise.

Think of it this way

Think of changing the map before measuring distances. Two towns may look far apart on a geographical map but close on a map of travel time. A learned representation similarly changes which observations look related. The limit is that a feature map is learned for a statistical task; its closeness does not necessarily explain a real-world cause or a universally meaningful similarity.

Worked example

Consider an original one-dimensional calculation using the feature transformation g(x) = x². This simple function is a stand-in for a learned feature map, not a trained deep network. Use the unit-variance, unit-length-scale base kernel k(a, b) = exp(−(a − b)² / 2).
For raw inputs −2 and 2, the untransformed kernel is exp(−16 / 2) = exp(−8), approximately 0.000335. After transformation, both features equal 4, so the transformed kernel is exp(0) = 1.
For inputs 1 and 2, the raw kernel is exp(−0.5), approximately 0.606531. Their transformed features are 1 and 4, giving exp(−4.5), approximately 0.011109.
The feature map therefore makes one pair much more closely related and another much less related. It also loses the sign of the input: −2 and 2 cannot be distinguished by this representation. If sign matters to the target, that collapse is a modeling problem. A flexible kernel cannot recover information that its input representation has discarded.

Illustrative case study

Fictional case: Meadow Analytics predicts a measured property from product images. Its team compares a neural regressor, a Gaussian process using fixed features, and a jointly learned deep kernel model. All are assessed on the same held-out production periods.
The deep kernel model performs well on common items but gives overconfident predictions on a new product family. Analysts inspect whether the representation maps unfamiliar items near familiar ones. They revise the evaluation and document limitations before operational use. The example illustrates a possible investigation, not a promised advantage or reported deployment result.

Common misconceptions and mistakes

Do not equate deep kernel learning with a deep Gaussian process. In the common formulation here, neural feature layers are deterministic given their fitted weights; deep Gaussian processes compose probabilistic function layers.
Do not assume that covariance-based uncertainty covers uncertainty about the feature extractor, distribution shifts, or omitted variables.
A learned representation can collapse different inputs together. Inspecting only the final predictive variance may conceal this issue.
Also distinguish joint learning from placing a Gaussian process on frozen pretrained features. Both can be useful, but their training procedures and flexibility differ. Neither should be declared superior without a relevant comparison.

Business use

For a business, the potential attraction is combining task-specific representation learning with a probabilistic prediction model. The decision should rest on measured usefulness: prediction quality, uncertainty quality, computational cost, and performance on new operating conditions.
Evaluate the feature extractor and kernel together. Retain a simple baseline, keep validation data independent of representation selection, and investigate regions where the model is confident but wrong. Record any approximation used in Gaussian process inference so that cost savings and their accuracy implications are visible during review.

Check your understanding

A feature map sends two distinct inputs to exactly the same feature vector. Under the unit-variance RBF base kernel, what is their kernel value?

Answer: It is 1 because their feature distance is zero. This establishes identical kernel features, not evidence that the original inputs are interchangeable for every task.

Related terms

Gaussian process
Neural network
Deep Gaussian process
covariance kernel
feature representation
marginal likelihood
inducing variable
predictive calibration

Learn next

Review how kernel distance affects Gaussian process predictions. Then compare raw-input kernels, kernels on frozen neural features, and jointly learned deep kernels. Check separately what each method learns and which uncertain quantities its predictive distribution actually integrates over.

Gaussian process

Sources

Wilson, Hu, Salakhutdinov, and Xing, Deep Kernel Learning, original formulation and learning method: https://proceedings.mlr.press/v51/wilson16.pdf Independent source review: Wilson and colleagues’ equation 5 and joint marginal-likelihood learning discussion were read.

Previous: Deep image prior Next: Deep lambertian networks