Must the networks have the same width?

No. They need representations of the same examples, but the numbers of features may differ.

Does CKA compare model weights?

The common representation comparison uses activations or features, not the parameter values themselves.

Is zero linear CKA a proof of independence?

No. It indicates no measured linear cross-covariance in the centered finite-sample calculation, which is weaker than a general independence conclusion.

AI Glossary / Probabilistic machine learning / Intermediate

Centered kernel alignment

Definition

Centered kernel alignment, or CKA, measures similarity between two sets of representations by comparing their centered similarity structures over the same examples. It is widely used to compare neural-network layers or trained models. It can compare representations with different numbers of features because the comparison can be expressed through example-by-example kernel matrices.

What it means

Two networks may encode the same images using different neuron counts or different coordinate systems. Comparing neuron number one in one model directly with neuron number one in another may therefore be uninformative.
CKA instead asks whether the two representations arrange the examples similarly according to a chosen kernel. The kernel produces a matrix describing similarities between pairs of examples; centering removes the corresponding mean effects before comparison.
A high score describes agreement under this measurement on the selected examples. It does not establish that the networks make identical predictions, have equal accuracy, or respond identically outside the comparison dataset.

How it works

Collect representations X and Y for exactly the same examples in matching row order. They may have different feature dimensions.
Choose kernels, such as linear inner products, and form the two Gram matrices K and L. Center each matrix using H = I − 11-transpose/n, giving K_c = HKH and L_c = HLH.
The usual centered alignment is their matrix inner product divided by the product of their Frobenius norms: sum(K_c × L_c) / (norm(K_c) × norm(L_c)). Multiplication in that sum is element by element. The denominator requires nonzero centered matrices.
For column-centered feature matrices and linear kernels, the same score is norm(X-transpose Y) squared divided by norm(X-transpose X) times norm(Y-transpose Y), using Frobenius norms. Linear CKA is invariant to orthogonal changes of feature coordinates and uniform nonzero scaling, but not arbitrary invertible transformations.

Three practical examples

A research team compares corresponding layers of networks trained from different random initializations.
An engineer compares a model before and after compression to see where its internal representation structure changes.
An analyst constructs a layer-by-layer similarity matrix between architectures with different hidden-layer widths, using a shared evaluation set.

Think of it this way

Think of comparing two maps by asking whether the same towns have similar relationships, even when one map uses a different orientation or overall scale. Individual map coordinates need not match directly.
The analogy has limits: CKA uses a specified kernel and centering procedure rather than every possible geographic relationship. Its invariances depend on the chosen version, and similar maps of one region do not establish agreement everywhere.

Worked example

Consider an original example with three matched observations and one feature per representation. Let X = (−1, 0, 1) and Y = (−2, 0, 2). Both columns have mean zero.
For single-feature linear CKA, the formula becomes (X dot Y) squared divided by (X dot X) times (Y dot Y). Here X dot Y = 4, X dot X = 2, and Y dot Y = 8. The score is 16 / (2 × 8) = 1.
The values are not identical, but Y is a uniform rescaling of X. Now compare X with Z = (1, −2, 1), which is also centered. X dot Z = −1 + 0 + 1 = 0, while Z dot Z = 6, giving CKA = 0 / (2 × 6) = 0.
These calculations concern a small linear example. The zero score does not prove general statistical independence, and the score of one does not prove that two complete prediction systems are interchangeable.

Illustrative case study

In a fictional project, Elm Vision compares an original image classifier with a compressed version. Several layers have high linear CKA on a shared image set, but evaluation still finds different predictions on a small group of difficult examples.
The team retains CKA as an internal diagnostic and reports predictive performance separately. It repeats the representation comparison on relevant subsets instead of interpreting one aggregate score as complete equivalence.
This invented case illustrates a distinction between representation similarity and task behavior. It makes no claim about a particular compression method or measured result.

Common misconceptions and mistakes

Do not compare matrices whose rows refer to different examples or differently ordered observations.
Do not forget centering when claiming to compute centered kernel alignment.
Do not interpret 0.9 as 90% accuracy or as a calibrated probability that two models are the same.
Do not assume that linear CKA ignores every transformation. Rescaling individual feature directions differently can change it.
Do not transfer linear-kernel invariance claims to a nonlinear kernel without checking its parameters. For an RBF kernel, scaling behavior depends on how bandwidth is chosen.
Finally, a constant representation can produce a zero denominator after centering; an undefined score should not be silently reported as ordinary similarity.

Business use

CKA can help a business-facing ML team investigate whether retraining, compression, or architectural changes alter internal representations. It is useful for locating changes that deserve further analysis.
Use a representative shared dataset and record preprocessing, kernel choice, and the estimator. Compare scores alongside task accuracy, subgroup behavior, and relevant robustness tests. A similarity diagnostic can guide investigation, but it does not by itself validate a replacement model or explain the cause of its errors.

Check your understanding

A representation is multiplied uniformly by three. What happens to its linear CKA with a fixed nondegenerate representation?

Answer: The score is unchanged. The numerator gains a factor of nine and the relevant denominator factor also gains nine, so they cancel.

Related terms

Gram matrix
Canonical Correlation Analysis
kernel matrix
representation similarity
Hilbert–Schmidt independence criterion
linear kernel
RBF kernel

Learn next

compute linear CKA for small centered matrices. Then compare a uniform rescaling with unequal rescaling of feature columns, and examine how the latter can alter the representation similarity score.

Sources

Primary source: Kornblith, Norouzi, Lee, and Hinton, Similarity of Neural Network Representations Revisited: https://proceedings.mlr.press/v97/kornblith19a/kornblith19a.pdf Publication record: https://proceedings.mlr.press/v97/kornblith19a.html The numerical vectors and fictional case are original illustrations.

Previous: Center-surround Next: Centering