Does the kernel include every layer?

It includes derivatives for the parameters designated as trainable; freezing layers changes that choice.

Is the NTK itself a neural network?

No. It is a kernel derived from a network and its parameterization.

Can different inputs have identical gradient features?

Yes. Distinct inputs alone do not ensure a nonsingular Gram matrix.

AI Glossary / Probabilistic machine learning / Advanced

Neural tangent kernel

Definition

The neural tangent kernel (NTK) measures how similarly a neural network's outputs at two inputs respond to small changes in its trainable parameters. It is formed from the inner product of output gradients. It connects parameter training with kernel methods and provides a way to study learning dynamics.

What it means

Two inputs can look different yet have aligned parameter gradients. An update driven by one can then move the other's prediction in a similar direction. Opposing gradients can produce opposing changes. The kernel describes this local coupling, rather than ordinary resemblance between raw inputs.
For a finite network, this coupling can change during training. Under particular parameter scaling, initialization and regularity assumptions, infinite-width theory yields a deterministic kernel that remains fixed over the training intervals considered. Applying that approximation to an actual model requires checking its suitability.

How it works

For a scalar output f_theta(x), collect derivatives with respect to all trainable parameters into g_theta(x). Define K_theta(x,x') = g_theta(x) dot g_theta(x'). Evaluating every pair in a dataset produces a Gram matrix. It is positive semidefinite because it is built from inner products; it need not be invertible.
Under continuous gradient flow, the chain rule relates prediction changes to this kernel and the loss derivatives. A discrete step uses a local approximation, with additional error possible when parameters move appreciably. Vector outputs require corresponding output-index blocks.
A practical investigation specifies which parameters are trainable, the loss normalization and parameterization. Changing these can change the kernel or its relationship to the update. Comparing a frozen initial kernel with the network's later behavior helps test the approximation.

Three practical examples

Training diagnosis: a research team compares prediction changes with those predicted by the initial NTK. Growing disagreement prompts investigation of changing gradients, step size and optimizer assumptions.
Model comparison: an analyst builds kernel baselines from two architectures on the same modest dataset. Held-out performance tests whether their initial local sensitivities capture useful distinctions.
Learning demonstrations: an instructor uses a two-parameter network to show why a valid tangent kernel need not remain constant. Students calculate both the approximate update and the actual update.

Think of it this way

Imagine several adjustable controls connected to multiple gauges. Each gauge has a sensitivity profile describing how every control affects it. Comparing two profiles reveals whether moving controls to adjust one gauge will also move the other. The NTK compares those profiles; the connections themselves may change as the controls move.

Worked example

Consider an original toy two-layer linear network f_(a,b)(x) = abx. It is linear in its input but not jointly linear in its two parameters. Initially a = 1 and b = 2. Its parameter gradient is (bx, ax), giving K(x,x') = (a squared + b squared)xx' = 5xx'.
For inputs 1 and 3, the initial kernel matrix has rows (5, 15) and (15, 45). It is positive semidefinite but singular: the second gradient is three times the first. Two distinct inputs therefore do not guarantee an invertible kernel matrix.
Now train on just x = 1 with target 0, loss L = f(1) squared / 2 and learning rate 0.1. The initial prediction is 2. The loss gradients with respect to a and b are 4 and 2. Simultaneous gradient descent gives a_new = 0.6 and b_new = 1.8, so the actual new prediction is 1.08.
Using the initial tangent approximation instead predicts 2 minus 0.1 times K(1,1) times 2 = 1. The difference of 0.08 comes from the product of the parameter changes. At the updated parameters, the kernel coefficient becomes 0.6 squared + 1.8 squared = 3.6, rather than 5. This small finite network illustrates both a changing NTK and finite-step approximation error; it is not an infinite-width experiment.

Illustrative case study

Fictional company Cedar Forecasting compares a neural regression model with its initial tangent-kernel approximation. Its analysts use identical training splits and record output scaling, trainable layers and learning rates. They evaluate both predictions on a separate validation set.
When their trajectories diverge, the team checks smaller steps and measures changes in the empirical kernel on a fixed subset. This separates a possible discretization problem from a poor frozen-kernel approximation. They retain the experiment as a diagnostic, rather than treating closeness to kernel behavior as the business objective. This is an illustrative scenario, not a reported deployment or measured result.

Common misconceptions and mistakes

An NTK is not simply the covariance of network outputs at random initialization; that is a different kernel construction. A finite network's NTK is not automatically constant. Positive semidefinite does not mean strictly positive definite or invertible. Kernel similarity need not be positive for every input pair.
Another mistake is treating the tangent approximation as a complete account of every trained network. Optimizers, parameter scaling and representation changes matter. Good agreement on training predictions also does not establish reliable predictions on new populations.

Business use

The NTK is mainly a research and diagnostic tool. It can help teams investigate whether added network complexity improves on an architecture-specific kernel baseline. Estimating full gradient features or storing all pairwise similarities can be expensive, so a representative subset may be more practical. Decisions should still consider validation error, operating cost and robustness to the intended deployment conditions.

Check your understanding

Why does the toy network's kernel coefficient fall from 5 to 3.6?

Answer: Both parameters change, and their squared values determine the coefficient.

Why does the tangent prediction differ from 1.08?

Answer: It omits the second-order product of the finite parameter changes.

What would test a frozen-kernel approximation?

Answer: Compare its predicted trajectory with actual training and measure kernel changes under matched conditions.

Related terms

Kernel method
Gradient descent
Gram matrix
Gaussian process
Feature learning
Kernel methods
Linearization
Infinite-width neural network

Learn next

Calculate output gradients for a small network, form its Gram matrix and check one training step. Then compare finite-network behavior with the assumptions behind infinite-width NTK theory.

Gram matrix

Sources

Jacot, Gabriel and Hongler, Neural Tangent Kernel: Convergence and Generalization in Neural Networks, 2018, especially the kernel definition and limiting-training assumptions: https://arxiv.org/abs/1806.07572 . Numerical example and fictional case are original teaching illustrations. Full primary paper independently consulted for the gradient kernel and Theorem 2 limiting-training assumptions.

Previous: Neural style transfer Next: Neural Turing Machine Network