How is SVD related to PCA?

Applying SVD to an appropriately centered data matrix can produce principal directions and scores. Uncentered SVD is not automatically the same analysis.

Can it handle complex matrices?

Yes. The factors use unitary matrices and conjugate transpose.

Does every software function return Σ as a matrix?

No. Many return a vector of singular values and a transposed or conjugate-transposed right factor, so the interface must be checked.

AI Glossary / Mathematics and optimization / Intermediate

Singular value decomposition

Definition

Singular value decomposition, or SVD, factors a matrix into orthonormal input and output directions together with nonnegative scale factors. In standard notation, A=UΣV*, where the diagonal entries of Σ are singular values and the star denotes conjugate transpose.

What it means

A data matrix can be viewed as a linear transformation. SVD describes that transformation through three stages: change input coordinates, scale along selected directions, then change output coordinates. For real matrices, these coordinate changes are orthogonal, and the star becomes ordinary transpose.

The decomposition exists for rectangular as well as square matrices, including rank-deficient matrices. It provides a common foundation for pseudoinverses, least-squares calculations, dimensionality reduction and low-rank approximation.

The components are mathematical combinations of the original coordinates. They do not automatically correspond to independent causes, named topics or meaningful business factors. Interpretation depends on the data representation and the problem.

How it works

For an m-by-n matrix, a full SVD has U of shape m-by-m, V of shape n-by-n and Σ of shape m-by-n. Only the main diagonal of Σ may be nonzero. A reduced form keeps k=min(m,n) columns in U and V and represents the scales with k singular values.

Another form writes A as a sum of outer products: A=Σᵢσᵢuᵢvᵢ*. Each term is a rank-one matrix, and the singular values are conventionally ordered from largest to smallest. Keeping only the first r terms yields a truncated SVD.

The truncated construction provides a best rank-at-most-r approximation in the Frobenius and spectral norms. The discarded singular values quantify the corresponding approximation error. This optimality concerns a specified matrix norm, not every downstream prediction task.

Singular vectors can have sign or phase ambiguities, and repeated singular values permit additional basis choices. Two implementations may therefore return different-looking vectors while representing the same decomposition. Compare reconstructed matrices or relevant subspaces rather than assuming matching signs are required.

Three practical examples

A document-analysis system uses a truncated SVD to construct a compact representation of a term-document matrix.
A calibration solver uses singular-value structure to calculate a minimum-norm least-squares solution.
An analyst examines the reconstruction error incurred when a matrix is approximated by a small number of components.

Think of it this way

Think of describing a transformation by first turning your coordinate grid, stretching along its axes and then turning the output grid. The separate steps make the overall operation easier to understand.

For rectangular matrices, the input and output spaces have different dimensions, so the picture requires adding or losing directions. The analogy also does not imply that the discovered axes have an obvious business meaning.

Worked example

Reconstructing and truncating a matrix

Let A have rows (3,3) and (1,−1). Choose U=I, Σ=diag(3√2,√2), and V with columns v₁=(1,1)/√2 and v₂=(1,−1)/√2. These columns are orthonormal.

Multiplying ΣVᵀ gives the first row (3,3) and second row (1,−1), so UΣVᵀ=A. The singular values are 3√2 and √2.

Keep only the first term. The rank-one approximation A₁ has rows (3,3) and (0,0). The residual A−A₁ has rows (0,0) and (1,−1), with squared Frobenius norm two. Its Frobenius norm is therefore √2, matching the discarded singular value.

The original squared Frobenius norm is 9+9+1+1=20. The retained term accounts for 18/20=90% of that squared magnitude. This does not mean it retains 90% of predictive usefulness: if the second row carries the task's distinguishing signal, losing it could be consequential. Nor is this percentage automatically explained variance unless the data and centering convention support that interpretation.

Illustrative case study

A fictional research archive builds a compressed representation of document features. Its team initially chooses a component count solely to retain a high percentage of squared matrix magnitude.

A reviewer checks retrieval examples involving rare technical terms and finds that the intended evaluation must include those cases explicitly. Components with smaller singular values are not guaranteed to be irrelevant to the archive's users.

The team compares reconstruction error with task-based retrieval performance and documents the chosen compromise. SVD supplies a mathematically controlled approximation, while the evaluation determines whether that approximation serves the intended use.

Common misconceptions and mistakes

SVD and eigenvalue decomposition are identical. SVD applies to rectangular matrices and uses distinct input and output directions.
A truncated SVD is best for every objective. Its optimality is tied to particular matrix approximation norms.
Every singular vector is uniquely determined. Signs, phases and repeated-value subspaces create ambiguity.
Small components are always noise. They may contain useful low-magnitude structure.
SVD automatically centers data. Centering is a separate preprocessing choice.

Business use

SVD helps teams reason about information compression and numerical sensitivity with explicit error measures. It can support efficient representations, but choosing a component count should reflect both computational limits and the intended task.

Document the original matrix, transformations, centering and scaling, retained rank and evaluation criteria. Distinguish full SVD from algorithms that compute only selected components or randomized approximations. A compact factorization saves storage only when its factor sizes and overhead are favorable compared with storing the original matrix.

Check your understanding

The worked rank-one approximation retains 90% of squared matrix magnitude. Does that prove it preserves 90% of the model's accuracy?

Answer: No. The percentage concerns a matrix norm. Predictive or retrieval accuracy depends on which information the task needs and must be evaluated separately.

Related terms

Learn next

Read singular value and principal component analysis. Reconstruct the worked matrix from its two outer-product terms, then identify exactly which row information is removed by retaining only the first term.

Sources

NumPy, numpy.linalg.svd: https://numpy.org/doc/stable/reference/generated/numpy.linalg.svd.html. Official factor shapes, return conventions and reconstruction relationships.

Aalto Dictionary of Machine Learning, SVD: https://aaltodictionaryofml.github.io/ADictML_Main.pdf. Source-list factorization. The explicit decomposition and task-interpretation example here are original illustrations.

Previous: Singular value Next: Singularity