Does supervised PCA need outcomes for new cases?

No. Outcomes guide training; the learned transformation is then applied to new predictors.

Is it the same as partial least squares?

No. They use outcome information through different constructions.

Can it miss useful predictors?

Yes, especially when marginal screening fails to detect joint or nonlinear predictive effects.

AI Glossary / Probabilistic machine learning / Intermediate

Supervised principal component analysis

Definition

Supervised principal component analysis uses outcome information to guide a lower-dimensional representation of predictors. Unlike ordinary PCA, which seeks directions of predictor variation without using the outcome, supervised approaches try to retain structure relevant to prediction. The name covers several formulations, so the specific selection rule or dependence objective should be stated.

What it means

The largest source of variation in a dataset may have little relationship to the quantity a model must predict. A production sensor may vary strongly with ambient conditions while a smaller coordinated signal indicates product quality. Ordinary PCA can prioritize the larger variation.

One influential approach, developed by Bair and colleagues, first screens predictors by their individual association with the outcome and then computes principal components of the retained predictors. Another formulation, developed by Barshan and colleagues, constructs directions using a dependence criterion involving the response. These are related supervised dimension-reduction ideas, but their algorithms are not interchangeable. The worked example below uses the screening approach.

How it works

For the screening formulation, center and otherwise preprocess predictors using training data. Calculate a suitable outcome-association score for each predictor, such as a standardized univariate regression score. Retain predictors exceeding a threshold in absolute score, compute PCA on that reduced matrix, and use one or more component scores in a predictive model.

Select the screening threshold and component count through a proper validation process. Every outcome-dependent screening step must be repeated within the training portion of each validation split. PCA and preprocessing must also be fitted without using the held-out portion when evaluating the complete training procedure.

At prediction time, apply the stored preprocessing and selected feature set to new observations, project them with the learned loadings, and use the fitted prediction model. New outcome values are not required. Screening can miss variables that matter only jointly or through nonlinear interactions, so the method's assumptions deserve explicit evaluation.

Three practical examples

1. Quality prediction: reduce many correlated sensor features after screening for association with a quality outcome. Validate across production periods rather than assuming random row splits represent future conditions.

2. Document scoring: construct a compact predictive representation from many text-derived features. Ensure that feature selection uses only labeled training examples and does not incorporate evaluation labels.

3. Operational forecasting: combine groups of correlated indicators into a few supervised components. Compare with regularized regression and ordinary principal component regression under the same chronological validation design.

Think of it this way

Think of building a summary of a large report for a particular question. Ordinary PCA summarizes the biggest differences in the report. A supervised version first asks which differences relate to the question, then constructs a compact representation. That focus can help prediction, but it can also overfit if the answer sheet for the test examples is consulted during summarization.

Worked example

Consider an original four-observation example with centered outcome y = (minus 1, minus 1, 1, 1). Let predictors X1 and X2 both equal y. Let X3 = (minus 4, 4, minus 4, 4). Use centered but otherwise unscaled predictors for this illustration.

X3 has squared norm 64, while X1 and X2 each have squared norm 4. X3 is orthogonal to both. The predictor cross-product matrix has rows (4, 4, 0), (4, 4, 0), and (0, 0, 64). Its leading eigenvector is the X3 direction, with eigenvalue 64, exceeding the correlated pair's eigenvalue 8. Ordinary PCA's first component is therefore unrelated to y in this toy.

Use the screening score Xj transpose y divided by the norm of Xj. The scores are 2 for X1, 2 for X2, and zero for X3. A stipulated threshold of 1 retains X1 and X2. Their first principal-component loading is (1, 1) divided by the square root of 2, up to sign. Its scores equal the square root of 2 times y.

Regressing y on those scores gives coefficient 1 divided by the square root of 2 and reproduces y in these four training observations. This is an algebraic illustration, not a validation result. Standardizing every predictor would change the ordinary-PCA comparison, and the threshold in a real application should be chosen without test-label leakage.

Illustrative case study

Fictional case: Willow Manufacturing wants a compact quality predictor from many correlated measurements. Analysts compare screened principal components, ordinary PCA regression, and a regularized regression baseline. They repeat feature screening inside each training fold and reserve a later production period for final evaluation.

The team checks whether selected features remain stable across batches and whether the components' predictive value survives changes in operating conditions. It does not interpret selected features as proven causes of quality differences. This fictional workflow reports no measured improvement and does not assume that the lowest-dimensional model is automatically the most useful.

Common misconceptions and mistakes

Supervised PCA is not a single universally standardized algorithm. It differs from simply adding labels to a plot after ordinary PCA. Selecting variables using the entire labeled dataset before cross-validation leaks outcome information. High univariate association is not proof of causality, while low univariate association does not rule out a useful interaction. Explain whether features were standardized because scaling affects the component calculation.

Business use

The practical benefit is focusing a compact representation on a prediction task rather than on variance alone. The tradeoff is additional dependence on labels, tuning, and validation design. Teams should record the formulation, preprocessing, association score, screening threshold, retained features, and component count. A simpler representation can aid maintenance, but prediction quality must be measured on genuinely unseen cases.

Check your understanding

Answer: 1.

Why does ordinary PCA select X3 first in the toy?

Answer: Its variance dominates the predictor matrix under the stated scaling.

Answer: 2.

Why does screening remove X3?

Answer: Its univariate outcome-association score is zero.

Answer: 3.

What must be repeated inside each validation training fold?

Answer: Outcome-dependent screening and the remaining fitted preprocessing and component steps.

Related terms

Principal component analysis
Principal component regression
Cross-validation
Data leakage
feature screening
partial least squares
dimension reduction

Learn next

Compare variance-based and outcome-based feature priorities on a small dataset. Then implement the complete screening-and-PCA pipeline inside validation folds and study alternative dependence-based supervised PCA formulations.

Sources

Bair and colleagues, Prediction by Supervised Principal Components, describes screening before PCA: https://web.stanford.edu/~hastie/Papers/spca.pdf

Barshan and colleagues, Supervised Principal Component Analysis, describes a dependence-based formulation: https://uwaterloo.ca/data-analytics/sites/default/files/uploads/documents/barshan_supervised_preprint.pdf

The four-observation calculation and Willow Manufacturing case are original; the case is fictional.

Independent review consulted Barshan Sections 3.4–5; Bair’s author-hosted PDFs were inaccessible, so its screening procedure was cross-checked in Section 3.4.

Previous: Supervised learning Next: Support vector machine