Does high-dimensional always mean d > n?

The supplied glossary emphasizes that case, but broader literature also studies dimension comparable to sample size.

Can an interpolating model generalize well?

In some settings, yes; it depends on data geometry, noise and the solution-selection rule, not training fit alone.

Does ridge regularization create new information?

No. It selects and shrinks coefficients according to a penalty, which can help estimation but introduces a modeling preference.

AI Glossary / Mathematics and optimization / Advanced

High-dimensional regime

Definition

The high-dimensional regime is a statistical setting in which model dimensionality is large relative to the amount of training information. The Aalto glossary emphasizes cases where effective model dimension exceeds sample size, such as linear regression with more features than observations. Broader statistical literature also uses the term when dimension and sample size are of comparable order. The defining relationship should therefore be stated rather than reduced to one universal numerical threshold.

What it means

High-dimensional does not simply mean that a dataset contains many rows or occupies a large file. The concern is how many adjustable directions must be learned from the available evidence. A thousand features with millions of independent observations presents a different problem from a thousand features with a hundred observations. Raw parameter count, matrix rank and effective dimension are related but distinct notions. Regularization, parameter sharing and data structure can reduce the model's practical flexibility, so counting stored weights alone does not fully characterize its statistical behavior.

How it works

When a linear model has d parameters and n observations with d > n, its design matrix cannot have full column rank. Multiple coefficient vectors may fit the same training values, and additional criteria are needed to choose among them. Regularization or a minimum-norm rule provides such a criterion, while sparsity or other structural assumptions can make learning feasible. These choices affect behavior on new data. High-dimensional analysis studies the joint effects of sample size, dimension, noise, geometry and estimation rules. It does not imply that every overparameterized model must generalize poorly or that interpolation is automatically safe.

Three practical examples

A text classifier can have many candidate token features relative to the number of labeled documents, requiring careful control of fitting and evaluation.
A scientific regression study can measure many variables on relatively few specimens and examine which structural assumptions support reliable estimation.
A machine-learning experiment can compare different solution-selection rules that fit the training data equally well but produce different predictions on new inputs.

Think of it this way

Think of adjusting many controls using only a few observations of the result. Several settings may explain what you have seen, yet disagree about an unseen situation. The analogy captures underdetermination but is incomplete: useful structure can connect the controls, and some directions may not matter for prediction. More controls do not always mean that successful learning is impossible.

Worked example

Consider a linear model with two training observations and three coefficients. Let the design matrix have rows (1, 0, 0) and (0, 1, 0), with targets (1, 2). Every coefficient vector β = (1, 2, t), for any real t, fits the training data exactly. The training mean squared error is zero for all of them. At a new feature vector (0, 0, 1), however, the prediction is t, which the training observations have not constrained. The minimum-Euclidean-norm interpolating solution chooses t = 0, but that is a selection rule, not evidence that the correct new target is zero. Now minimize ||Xβ − y||² + λ||β||² with λ > 0 and all three coefficients penalized. The unique solution is β = (1/(1 + λ), 2/(1 + λ), 0). At λ = 1 it becomes (0.5, 1, 0), with training squared-error sum 0.25 + 1 = 1.25. Regularization gives up exact fit and chooses a unique vector, but the example supplies no target for the unseen third direction, so it cannot establish which rule predicts better there. The missing information remains a statistical issue rather than an algebraic inconvenience.

Illustrative case study

In a fictional document-classification pilot, Birch Records has many candidate text features and a limited labeled collection. The team compares regularization strengths within a controlled validation procedure and checks whether near-duplicate documents cross the split. Analysts avoid treating zero training error as a success criterion. They also investigate whether feature construction reflects information available when predictions will be made. The case illustrates evaluation discipline, not a claim that a particular feature reduction or penalty necessarily improves the model.

Common misconceptions and mistakes

High dimensionality is relative to available information, not a fixed feature count. More parameters than observations does not guarantee failure, and fewer parameters does not guarantee generalization. Raw observation counts can overstate information when records are strongly dependent or duplicated. Effective dimension has several technical definitions and should not be used as an unexplained synonym for stored parameter count. A unique regularized solution is not automatically unbiased or accurate. Finally, results about a particular high-dimensional probabilistic model should not be generalized to every neural network or dataset.

Business use

Report sample size, feature construction and the dimension notion used. Examine rank, dependence and data quality rather than relying on a parameter-to-row ratio alone. Choose regularization or structural assumptions based on the task and assess them through a suitable validation design. Keep feature selection inside the training process to avoid leaking evaluation information. Evaluate stability across reasonable data splits and inspect uncertain predictions in poorly supported directions. For stakeholders, explain which aspects of the fitted model are supported by observations and which depend more strongly on modeling choices.

Check your understanding

Why can every worked coefficient vector have zero training error while making a different prediction at (0, 0, 1)?

Answer: The training matrix never measures the third coefficient. Changing it leaves training predictions unchanged but changes the new prediction.

Related terms

Effective dimension
Regularization
Generalization
overparameterization
matrix rank
interpolation

Learn next

add a third observation with feature vector (0, 0, 1) and see how it constrains t. Then make that observation noisy and compare exact fitting with regularization. Explain why uniqueness, training fit and predictive accuracy answer different questions.

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, high-dimensional regime, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; Hastie, Montanari, Rosset and Tibshirani, Surprises in High-Dimensional Ridgeless Least Squares Interpolation, https://arxiv.org/abs/1903.08560 . The paper's abstract was reviewed for its scope and terminology; no detailed theorem is asserted here. The underdetermined and ridge examples are original.

Previous: Hierarchical variational posterior Next: High-dimensional statistics