Can effective dimension be noninteger?

Yes. Fractional contributions are central to the spectral definition used here.

Is smaller always better?

No. Excessive restriction can underfit, and predictive usefulness still needs evaluation for the intended task.

Does more data always increase the reported number?

Not under every definition and normalization. Its behavior depends on how the matrix, fitted model and regularization scale change with the data.

AI Glossary / Probabilistic machine learning / Advanced

Effective model dimensionality

Definition

Effective model dimensionality describes how many independent directions of model flexibility are meaningfully determined or used under a specified dataset, model and regularization scheme. It can differ substantially from the raw number of stored parameters. The exact numerical definition must be stated because several related measures use this terminology.

What it means

A model may contain many coefficients without the data distinguishing all their combinations. Some directions are strongly supported; others are weakly identified, redundant or heavily constrained by a prior or penalty. Counting every coefficient equally misses this distinction.

Here the focus is a common spectral effective-dimension measure used in Bayesian and regularized modeling. It assigns fractional contributions to parameter-space directions according to their strength relative to a regularization scale. It is not simply the number of input columns, a neural network's embedding width or the intrinsic dimension of the observed data manifold. Those answer different questions.

How it works

For a positive semidefinite matrix A with eigenvalues lambda_i and a positive scale z, one common measure is the sum of lambda_i/(lambda_i + z). The matrix may describe a design, kernel or suitable local curvature, depending on the model and interpretation.

Large eigenvalues compared with z contribute nearly one; small ones contribute nearly zero. With these nonnegative eigenvalues, each contribution lies between zero and one. In a Gaussian linear model, related quantities express how much the data determine parameter directions relative to the prior.

For nonlinear models, a local curvature interpretation is approximate. The matrix definition, loss normalization, parameter scaling and chosen z matter. Arbitrary indefinite Hessians do not inherit the simple fractional-count interpretation without additional treatment. The number is meaningful only together with this specification.

Three practical examples

1. In ridge regression, an analyst studies how stronger regularization reduces the effective flexibility associated with weak feature directions while leaving the stored coefficient count unchanged.

2. A kernel-model researcher compares effective dimensions computed at several regularization scales. The same kernel matrix can produce different values because the scale changes the question.

3. A neural-network experiment examines local curvature to investigate why parameter count alone poorly describes fitted-model behavior. The diagnostic supplements, rather than replaces, direct generalization testing.

Think of it this way

Think of a control panel with many knobs, some of which move the same mechanism and others of which barely affect the output. The printed knob count overstates the number of distinct adjustments available in practice. The limit of this analogy is that effective dimension can be fractional and depends on data and regularization; it is not a permanent inventory of physically active knobs.

Worked example

Take a positive semidefinite matrix with eigenvalues 9, 1 and 0.1. At z = 1, the three contributions are 9/10 = 0.9, 1/2 = 0.5 and 0.1/1.1, approximately 0.0909. Their sum is approximately 1.4909, although the matrix has three dimensions and full rank.

Now keep the matrix fixed and use z = 9. The contributions become 9/18 = 0.5, 1/10 = 0.1 and 0.1/9.1, approximately 0.0110. The total falls to approximately 0.6110.

Nothing was deleted from the model. The count changed because the reference scale changed. All three eigenvalues remain positive, so algebraic rank remains three. As z approaches zero from above, each positive eigenvalue's contribution approaches one and this example's total approaches three.

This calculation illustrates a defined matrix statistic. It does not supply a prediction error, identify which individual coefficients can be removed or establish that the smaller number corresponds to a better fitted model.

Illustrative case study

Fictional case: Forecast Orchard compares sales regressions containing many overlapping seasonal indicators. Management initially judges complexity only by coefficient count. The analyst computes an effective-dimension diagnostic across several penalty strengths and explains why correlated indicators do not behave like equally independent degrees of freedom.

The team checks forecast error on future periods and examines whether heavier shrinkage erases genuine seasonal changes. It records feature scaling and the penalty definition alongside each diagnostic. A lower dimension becomes one piece of evidence about the fit, rather than an automatic approval criterion. No actual model-performance improvement is claimed.

Common misconceptions and mistakes

Effective dimensionality is not universally equal to rank. Rank gives a binary distinction between zero and nonzero directions, whereas this spectral measure weights their strengths continuously.

A low value does not prove accuracy, fairness, interpretability or small deployment cost. A large network may still require substantial memory and computation even if a particular effective-dimension diagnostic is low.

Do not compare numbers computed with different matrices or scales as though they were directly interchangeable. Rescaling a loss can rescale curvature and change the result unless the regularization reference is adjusted consistently. Nor should a fractional value be rounded and interpreted as permission to delete that many parameters.

Business use

This concept helps teams discuss model complexity more carefully when raw parameter count is misleading. Use it to understand the effect of regularization, redundancy or local information, while preserving operational measurements such as latency and memory use.

A useful report names the definition, data, matrix approximation and regularization scale, then pairs the result with held-out performance and uncertainty checks. For very large models, numerical spectral approximations can introduce additional error. Analysts should state that approximation rather than presenting a many-decimal answer as exact structural truth.

Check your understanding

If a fourth eigenvalue of exactly zero is added to the worked matrix, how much does it contribute when z is positive?

Answer: Zero, because 0/(0 + z) = 0. The nominal matrix size increases, but this effective-dimension measure does not.

Related terms

Effective degrees of freedom
Regularization
Eigenvalue
Hessian
matrix rank
posterior contraction
model complexity
Bayesian model selection

Learn next

Compare algebraic rank with the spectral sum across several z values. Then study ridge regression's fitted-value matrix and Bayesian linear regression to see when effective-dimension interpretations become exact.

Sources

Primary sources: Murphy, Probabilistic Machine Learning: Advanced Topics, Section 17.4.3, page 667, equation 17.36: https://github.com/probml/pml2-book/releases/latest/download/book2.pdf ; Maddox, Benton and Wilson, Section 2.3: https://arxiv.org/pdf/2003.02139 . Full sourcebook downloaded and relevant passage read independently, resolving earlier access and terminology uncertainty. The spectral definition matches. Earlier MacKay full-PDF access limitation remains; examples are original.

Previous: Effective dimension Next: Effective sample size