How does PCR differ from partial least squares?

Standard PCR chooses directions using predictor variation alone, while PLS uses information about the outcome when forming components.

Can all components be retained?

Yes. With a full-rank transformation and corresponding ordinary least-squares setup, retaining the full predictor subspace reproduces the original least-squares fit.

Should every predictor always be standardized?

No. The decision depends on units and the question, but it must be made deliberately because it affects PCA.

AI Glossary / Statistics and probability / Advanced

Principal component regression

Definition

Principal component regression, abbreviated PCR, predicts an outcome using selected principal-component scores derived from the predictors. The usual procedure first applies principal component analysis to the predictor matrix and then regresses the outcome on a chosen number of leading components. These components summarize directions of predictor variation. PCA does not use the outcome to choose those directions, so retaining the largest-variance components does not necessarily retain the directions most useful for prediction.

What it means

PCR can reduce dimensionality and handle strongly correlated predictors by replacing the original variables with orthogonal combinations. Keeping fewer components can stabilize estimation when many original directions are noisy or poorly determined. The trade-off is that discarded directions may contain useful outcome information. Explained variance in the predictors is not the same as explained variation in the target. A model retaining 95% of predictor variance can still omit a small-variance direction that is crucial for forecasting the outcome.

How it works

Fit any required centering and scaling using training data. Apply PCA to the transformed predictors, choose the retained components and fit a regression model on their scores. Select the component count using a validation procedure aligned with the prediction task, performing preprocessing and PCA within each training fold. For new observations, use the same fitted transformations and component loadings before applying the regression. Evaluate prediction against suitable baselines. Scaling is substantive because changing feature units can change which directions PCA treats as high variance; record whether features were standardized and why.

Three practical examples

Sensor modeling: predict an outcome from many correlated instrument measurements using a smaller set of component scores.
Spectral analysis: reduce a large set of related wavelength measurements before regression, with component selection based on predictive validation.
AI feature modeling: compare PCR with regularized regression when feature dimensions are numerous or redundant, while checking whether low-variance directions carry target information.

Think of it this way

Think of compressing a collection of measurements before making a prediction. The compression preserves the largest visible patterns in the inputs, but the clue needed for the answer may be a small detail. Good compression of inputs and good prediction of outcomes are related goals, not identical ones.

Worked example

Consider an original centered dataset with predictor columns x1 = (-2,-2,2,2) and x2 = (-1,1,-1,1). Their cross-product is zero, and their empirical variances using denominator 4 are 4 and 1. Without standardizing, the first principal component follows x1 and explains 4/(4+1) = 80% of predictor variance. Now set the target to y = 3x2 = (-3,3,-3,3). The cross-product of x1 with y is zero. PCR retaining only the first component fits zero predictions for this centered target, giving mean squared error 9. Keeping both components allows an exact fit in this deliberately noiseless example. This shows that the low-variance direction can contain all the target signal. It is an algebraic construction rather than evidence about real data. Standardizing these predictors would change the PCA variance structure, illustrating why preprocessing must be part of the stated method.

Illustrative case study

In a fictional case, Stone Sensors selects enough components to explain most input variation and assumes the resulting regression must be accurate. Held-out performance is disappointing. Analysts find that a subtle measurement contrast associated with the target lies outside the retained subspace. They compare component counts through proper validation and benchmark against ridge regression and partial least squares. The final choice is based on prediction quality and stability, with the preprocessing and component loadings saved for deployment.

Common misconceptions and mistakes

PCA is unsupervised with respect to the target, even when its scores later enter a supervised regression. High explained predictor variance does not guarantee low prediction error. Fitting PCA to the entire dataset before cross-validation leaks information about held-out inputs into model development. Components are orthogonal in the fitted training representation, but their interpretation still depends on preprocessing and sampling. Component signs are arbitrary and can reverse without changing the underlying subspace or predictions when regression coefficients are transformed consistently. PCR also does not reveal causal effects of the original predictors.

Business use

Businesses should consider PCR when compression or predictor correlation creates a practical modeling problem, then test whether it helps the actual target. Compare against simpler and regularized alternatives and examine stability across samples. Save centering values, scaling values, loadings and regression coefficients as one reproducible workflow. If stakeholders need original-variable explanations, make clear that each component mixes variables and that interpreting a component coefficient may be less intuitive than reading an ordinary regression coefficient. Monitor deployment inputs because the learned variance structure may change over time.

Check your understanding

The first component explains 80% of predictor variance. Does that imply it explains 80% of outcome variation?

Answer: No. Predictor variance and outcome predictiveness are different quantities; the worked example makes their distinction explicit.

Related terms

Learn next

Learn how PCA depends on scaling, then compare component selection based on predictive validation with selection based only on explained predictor variance.

Sources

Scikit-learn, Principal Component Regression vs Partial Least Squares Regression: https://scikit-learn.org/1.8/auto_examples/cross_decomposition/plot_pcr_vs_pls.html Scikit-learn, PCA documentation: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

Previous: Principal component analysis Next: Principal curve