Is this the same as k-nearest neighbors?

No. It compares against class representatives rather than voting among nearby individual training examples.

What does soft-thresholding do to a negative standardized difference?

It reduces its absolute magnitude toward zero while preserving the sign unless the result becomes zero.

Can every feature be removed?

A sufficiently strong threshold can erase all class deviations, making class distinctions depend on remaining conventions such as priors and tie handling.

AI Glossary / AI and machine learning / Advanced

Nearest Shrunken Centroid

Definition

Nearest shrunken centroid is a classification method that moves class-average feature values toward the overall average before assigning a new example to a class. Soft-thresholding standardized class differences can remove features that no longer distinguish any class.

What it means

A nearest-centroid classifier represents each class by its average feature vector and assigns a new observation according to its closeness to those representatives. With many noisy measurements and limited data, small differences between class averages can be unreliable.

Shrunken centroids reduce those differences. The shrinkage acts on standardized deviations from the overall centroid, not simply by subtracting a constant from every raw class mean. Small deviations become zero, while larger deviations are reduced in magnitude.

When a feature's deviations become zero for every class, all class centroids share the overall mean on that feature. It then contributes equally to class comparisons and can be omitted. The result combines classification with a form of feature selection.

How it works

Calculate the overall centroid, class centroids and pooled within-class variability. The original method standardizes each class-minus-overall difference using a scale that includes within-class standard deviation, a stabilizing term and a class-size factor.

Apply soft-thresholding to each standardized difference: retain its sign and reduce its magnitude by a threshold, stopping at zero. Convert the surviving differences back into shrunken class centroids.

Classify new observations using standardized distances to those centroids, with class-prior adjustments where the formulation requires them. Choose shrinkage using appropriate validation rather than maximizing the number of discarded features.

Software implementations can differ in scaling and prior details. The core idea is shrinkage of class deviations, so a short implementation description should not be mistaken for a complete mathematical specification.

Three practical examples

A researcher classifies high-dimensional measurements while reducing weak class differences through centroid shrinkage.
An analyst inspects which features retain nonzero deviations for at least one class.
A modeling team compares a simple nearest-centroid baseline with several shrinkage thresholds using held-out classification error.

Think of it this way

Think of drawing a typical profile for each group, then fading differences that are too small to trust. Stronger contrasts remain visible, while faint contrasts disappear into the common background.

The analogy describes shrinking differences toward an overall profile. The method uses numerical standardization and a threshold, not a subjective judgment that every small raw difference is unimportant. A small but stable measurement difference may be meaningful after scaling.

Worked example

Consider a simplified two-class example with overall centroid (10, 20). Class A has centroid (12, 20.2), and class B has centroid (8, 19.8).

For illustration, suppose the factors used to standardize these class deviations are all one. The standardized deviations are therefore A: (2, 0.2) and B: (−2, −0.2). Apply threshold 0.5.

Soft-thresholding gives A: (1.5, 0) and B: (−1.5, 0). The reconstructed centroids are (11.5, 20) and (8.5, 20). Feature 2 now has the same centroid value for both classes and contributes no difference between their distance scores.

With equal class priors and a common distance scale, a new point (11, 21) has unscaled squared distances 1.25 to A and 7.25 to B, so A is selected. This is deliberately a unit-scale demonstration; an actual fit must calculate the variability and class-size factors rather than assume them.

Illustrative case study

The fictional BeechMeasurements team classifies machine operating modes from many sensor summaries. Its nearest-centroid baseline uses every measurement, including several with unstable class averages.

The team tests centroid shrinkage and records both classification error and the retained feature set across validation folds. It notices that some weak features appear and disappear depending on the training sample, so it avoids treating one selected set as definitive evidence about the machinery.

It checks the reduced classifier on held-out operating periods. This invented case illustrates prediction and selection stability, with no measured accuracy improvement or claim that selected sensors establish causal mechanisms.

Common misconceptions and mistakes

Every class centroid is shrunk toward the origin. The class deviations are shrunk toward the overall centroid after standardization.
A zero deviation for one class removes the feature entirely. Removal requires the feature to lose its distinguishing deviations across all classes.
More shrinkage necessarily improves classification. Too much can erase useful class differences and leave little beyond prior preferences.

Business use

Nearest shrunken centroids can provide a relatively compact classifier and a readable set of class profiles. It is especially relevant when many measurements compete for limited training evidence.

Evaluate preprocessing, variability estimates and threshold selection within the validation procedure. Inspect class imbalance and prior handling. Check whether one centroid adequately represents each class; a class with several separated subgroups may not be well summarized by a single mean. Treat retained features as predictive model components rather than causal discoveries. Historical research applications do not substitute for validation in a new domain.

Check your understanding

A feature's shrunken centroid is 7 for every class. Does that feature help distinguish those classes through the centroid-distance comparison?

Answer: No. Its distance contribution is the same for every class and therefore cancels when comparing their scores.

Related terms

Feature Selection
Shrinkage
Pooled variance
Regularization
Nearest centroid
soft-thresholding
class prior
high-dimensional classification

Learn next

Read nearest-centroid classification and soft-thresholding. Repeat the worked example with different thresholds, then explain why standardized deviations should be examined before comparing raw feature differences.

Sources

Scikit-learn, Nearest Shrunken Centroid: https://scikit-learn.org/stable/modules/neighbors.html#nearest-shrunken-centroid

Robert Tibshirani, Trevor Hastie, Balasubramanian Narasimhan and Gilbert Chu, original 2002 paper: https://www.uvm.edu/~rsingle/stat380/F04/papers/Tibshirani%2BPNAS-2002_ShrunkenCentroids.pdf

The original paper's standardization, overall-centroid reference and soft-thresholding equations were reviewed. The unit-scale worked example is original and is not presented as a fitted software result.

Previous: Nearest Neighbors Regression Next: Necessary condition analysis