Does PCA require labeled outcomes?

No. Standard PCA finds variation in the input features without using a target label.

Are the components independent?

Their scores are uncorrelated under the fitted covariance description, but uncorrelated does not generally mean statistically independent.

Can PCA capture any nonlinear shape with a few components?

No. Ordinary PCA uses linear directions. Nonlinear structure may require different methods or may remain poorly represented by a small linear projection.

AI Glossary / Data science and analytics / Intermediate

Principal component analysis

Definition

Principal component analysis, or PCA, transforms numerical variables into new, mutually perpendicular directions ordered by how much variation they capture. Keeping only the leading components provides a lower-dimensional representation of the data, usually with some information loss.

What it means

A dataset may contain several measurements that move together. PCA looks for combinations of those measurements that summarize their shared variation. Each component is a weighted linear combination of the input variables after the chosen centering and scaling.

The first component captures the greatest available variance. The next captures the greatest remaining variance subject to being perpendicular to the first, and so on. The component scores locate observations along these new directions.

PCA is unsupervised: ordinary PCA does not use an outcome label to decide what matters. A direction with high variance is not automatically the best predictor, the most meaningful business factor or the cause of an observed pattern. Its usefulness must be assessed for the intended task.

How it works

Arrange observations as rows and numerical features as columns. Standard PCA centers each feature by subtracting its mean. Scaling features to comparable units is a separate choice, often important when raw units or variances differ substantially.

The principal directions can be calculated using singular value decomposition of the centered matrix or an eigendecomposition of its covariance matrix. The associated eigenvalues describe variance along the directions. Explained-variance ratios express each component's share of the total variance in the analyzed representation.

Choose how many components to retain, then project observations onto those directions. For predictive modeling, fit preprocessing and PCA only on training data and apply the learned transformation to evaluation data.

Inspect the feature weights and sensitivity to outliers and scaling. Component signs can be reversed without changing the underlying axis, and tied variances can make individual directions non-unique within a subspace.

Three practical examples

A sensor-analysis team compresses several correlated measurements into a smaller set of components for exploratory plots.
An analyst examines whether many questionnaire measurements share broad patterns of variation, while avoiding unsupported causal labels for the components.
An AI developer tests PCA as a preprocessing step and checks whether a smaller representation preserves useful predictive performance on held-out cases.

Think of it this way

Think of rotating a camera to view a long, thin object along directions that reveal its main extent. A carefully chosen view can capture much of its shape in fewer dimensions.

The analogy is limited because PCA operates on numerical data and uses variance, not visual interest. A small feature hidden by the chosen view may still be important to the task, just as a low-variance direction may contain useful predictive information.

Worked example

Take three invented two-feature observations: (1, 1), (2, 2) and (3, 3). The mean is (2, 2), so the centered observations are (-1, -1), (0, 0) and (1, 1).

Every point lies on the diagonal direction (1, 1). A unit vector along it is (1, 1) divided by the square root of 2. Projecting the centered points onto that direction gives scores of minus the square root of 2, 0 and the square root of 2.

The perpendicular direction, (1, -1) divided by the square root of 2, has zero scores for all three points. Therefore, one component captures all observed variance in this idealized example. Multiplying a score by the first direction and adding the mean reconstructs the original point exactly.

Real datasets usually have variation in several directions. Discarding components then loses information; a high explained-variance percentage does not guarantee that the discarded information is irrelevant.

Illustrative case study

Illustrative fictional case: Seabrook Services applies PCA to customer records containing annual spending and monthly visit counts. The first component is dominated by spending, whose numerical scale is much larger.

The analysts compare the raw-unit and standardized analyses and explain the different questions they represent. They do not automatically declare one correct: preserving absolute monetary variation and giving features comparable variance are different analytical choices.

Before using the reduced features in a prediction model, they fit the transformation within the training workflow and evaluate it on untouched data. This fictional case illustrates scaling and evaluation decisions, not a reported improvement in customer prediction.

Common misconceptions and mistakes

Do not confuse PCA with selecting a subset of original columns. Components are combinations, so their meaning can be less direct.

Do not equate variance explained with predictive accuracy or causal importance. Avoid assuming that a standard PCA implementation automatically standardizes every feature. Do not interpret a component's sign as intrinsically positive or negative in meaning, and do not ignore the effect of a few extreme observations on the directions.

Business use

PCA can help with exploration, compression and some modeling workflows when numerical features are redundant or high-dimensional. Begin with a clear reason for reducing dimension and compare the result with simpler alternatives.

Record centering, scaling, retained components and training data. For a business audience, explain the transformation and its information loss without giving components unsupported names such as “customer value” merely because the label sounds plausible. If downstream decisions depend on rare patterns, evaluate those cases explicitly rather than relying only on the total variance retained.

Check your understanding

Two components retain 95% of the input variance. Does that prove they retain 95% of the information needed to predict a target?

Answer: No. Variance is measured in the inputs, not by relevance to the target. Predictive usefulness needs separate evaluation, including information in discarded directions.

Related terms

Learn next

Read covariance and singular value decomposition. Plot the three-point worked example, center it and project it onto diagonal and perpendicular axes. Then examine how changing one feature's units affects a PCA analysis with and without standardization.

Sources

NIST, principal components and dimensionality reduction: https://www.itl.nist.gov/div898/handbook/pmc/section5/pmc55.htm

NIST, properties of principal components: https://itl.nist.gov/div898/handbook/pmc/section5/pmc551.htm

Scikit-learn, PCA centring, component directions and explained variance: https://scikit-learn.org/1.0/modules/generated/sklearn.decomposition.PCA.html

Scikit-learn, avoiding preprocessing leakage: https://scikit-learn.org/1.7/common_pitfalls.html

The diagonal data and customer-analysis case are fictional teaching examples.

Previous: Prime Attribute Next: Principal component regression