Does keeping 95% of variance mean keeping 95% of useful information?

No. Variance is one criterion and may not align with the task.

Can reduction improve a model?

Sometimes, but it can also remove useful signals; compare suitable held-out results.

Are distances on a two-dimensional plot always meaningful?

No. Their interpretation depends on the method and which relationships it preserves.

AI Glossary / Data science and analytics / Beginner

Dimensionality reduction

Definition

Dimensionality reduction represents data using fewer features or coordinates than the original representation. It can support visualization, compression, noise management, or more efficient modeling. Some methods select existing features; others construct new combinations or learned representations. Reducing the number of dimensions usually involves a choice about which information to retain, and the best choice depends on the intended task.

What it means

A record may contain hundreds of measurements even when many describe related aspects of the same underlying behavior. A smaller representation can make those relationships easier to explore or process. However, fewer dimensions do not automatically mean better data. A feature that explains little overall variation may still matter for a rare but important outcome. A two-dimensional picture is a projection of a richer dataset, so its apparent clusters and distances need careful interpretation.

How it works

An analyst chooses a method and a criterion suited to the purpose. Principal component analysis constructs orthogonal directions that capture successive amounts of variance in appropriately prepared data. Other methods seek different relationships, such as local neighborhoods or a representation useful for reconstruction. The transformation is fitted, the number of dimensions is chosen, and the reduced representation is evaluated. When reduction forms part of a predictive pipeline, fit it using training data and apply the learned transformation consistently to held-out data. Scaling and missing-value handling must also be defined.

Three practical examples

Exploratory visualization: a researcher projects high-dimensional document vectors into two dimensions to inspect candidate patterns, then checks those patterns against the original records.
Sensor modeling: an engineering team compresses correlated measurements and evaluates whether the reduced representation preserves useful predictive performance.
Storage and retrieval experiments: a team compares smaller vector representations with original vectors, measuring both resource use and the quality of retrieved results.

Think of it this way

Think of making a map from a landscape. The map can be easier to carry and interpret because it keeps selected relationships and omits others. A road map and a contour map serve different purposes. Likewise, a reduced representation preserves what its method prioritizes; it cannot be assumed to preserve every distance, category, or detail useful to another task.

Worked example

Consider three records with two measurements each: (1, 2), (2, 4), and (3, 6). Every second measurement is exactly twice the first. Although the records have two coordinates, they lie on one straight line. In this special example, retaining the first coordinate alone is sufficient to reconstruct the second by multiplying by two.
Now change the records to (1, 2.1), (2, 3.9), and (3, 6.2). The same one-coordinate reconstruction gives (1, 2), (2, 4), and (3, 6), leaving second-coordinate errors of 0.1, minus 0.1, and 0.2. Whether that loss is acceptable depends on the task. If tiny deviations identify a fault, discarding them could be harmful. A PCA projection would choose a fitted direction rather than simply retain the first coordinate; the example illustrates redundancy and reconstruction loss without pretending to compute that fitted PCA solution.

Illustrative case study

In a fictional maintenance project, many sensors move together because they respond to the same operating load. The team tests a reduced representation to simplify a failure-prediction model. It separates training and evaluation periods, fits preprocessing only on the training period, and compares missed failures as well as overall accuracy. A compact representation that preserves most variance still requires scrutiny if a low-variance sensor carries an early warning signal. The team documents the tradeoff and retains the original measurements for investigation. The scenario illustrates evaluation of information loss, not a claim that compression improved real equipment reliability.

Common misconceptions and mistakes

Dimensionality reduction is not always feature selection: new components can combine many original variables. Explained variance is not the same as predictive accuracy or business value. A visualization method optimized for local neighborhoods may distort global geometry, so separation on a plot does not establish a true operational category. Fitting reduction on the full dataset before a train-test split can leak information. Standardizing features is a modeling choice, not a universal requirement, and software that centers features may not also scale them.

Business use

A business should define what a smaller representation is expected to improve: response time, memory use, communication, or model performance. Measure the benefit alongside information loss and the ability to explain results. Preserve the transformation, preprocessing settings, and model version so future records are represented consistently. Test important subgroups and unusual cases, not just an average reconstruction score. If a reduction is used only for a chart, make that purpose explicit so readers do not mistake a convenient display for a validated segmentation model.

Check your understanding

A reduced representation removes a small sensor variation that predicts rare failures. Is high retained variance enough to accept it?

Answer: No. Evaluate the failure-prediction task and its error costs. A low-variance signal can be valuable even when it contributes little to total variation.

Related terms

Learn next

Start with correlated measurements and geometric projection, then learn PCA and the role of scaling. Compare reconstruction error with downstream predictive performance. Study visualization methods next, paying attention to local and global relationships, and practice fitting all learned preprocessing within a proper training pipeline.

Training Pipeline

Sources

Scikit-learn, decomposition and PCA: https://sklearn.org/stable/modules/decomposition.html . Scikit-learn, t-SNE reference: https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html . Inria scikit-learn course, dimensionality reduction of text data: https://inria.github.io/scikit-learn-mooc/python_scripts/dimred_text.html . Primary project documentation and course excerpts were consulted. The numerical and business examples are original illustrations. Editorial review remains pending.

Previous: Dimensionality Next: Dinitz