AI Glossary / Mathematics and optimization / Advanced
Positive semi-definite (psd)
Definition
A real symmetric matrix is positive semi-definite, abbreviated psd, when its quadratic form is nonnegative for every vector. Formally, Q is psd if xᵀQx ≥ 0 for all x. Unlike positive definiteness, this permits a nonzero vector to have quadratic value zero. Such directions often represent redundancy, invariance, or a lack of variation in a model.
What it means
Positive semidefiniteness describes how a matrix acts on whole directions, not whether its individual entries are nonnegative. It allows flat directions but excludes directions with negative quadratic curvature.
The distinction matters in AI because Gram matrices, covariance matrices, and graph Laplacians naturally have this property. They may nevertheless be singular. An algorithm that needs an invertible positive-definite matrix cannot automatically treat every psd matrix as interchangeable with one.
This article uses the real symmetric convention. Complex-valued counterparts use Hermitian matrices and conjugate transposes.
How it works
A real symmetric matrix is psd exactly when all its eigenvalues are nonnegative. Zero eigenvalues identify nonzero directions in its nullspace. A matrix of the form BᵀB is always psd because xᵀBᵀBx equals the squared length of Bx.
For a quadratic objective with Hessian Q, positive semidefiniteness gives convexity. It does not by itself guarantee a unique minimizer or even that a minimum exists when an arbitrary linear term is included. The linear term must also be compatible with any flat directions.
The notation A ≥ B can be ambiguous for matrices. In the semidefinite order, often written A ⪰ B, it means A − B is psd. That is a comparison of quadratic forms in every direction, not a comparison of corresponding entries.
Three practical examples
1. Covariance modeling: describe nonnegative variance for every linear combination of features, including zero variance when features have an exact linear relationship.
2. Kernel methods: use a kernel whose finite Gram matrices are psd, enabling the associated inner-product interpretation.
3. Graph smoothing: penalize differences between connected node values using a graph Laplacian, while recognizing that a constant signal on a connected component has zero smoothness penalty.
Think of it this way
Think of a trough whose surface never curves downward but remains flat along its length. Movement across the trough has a cost, while movement along the flat direction may have none. This illustrates a singular psd quadratic. Not every psd matrix has a flat direction: positive-definite matrices are also psd and form the stricter case.
Worked example
Take Q with rows (1, −1) and (−1, 1). For a vector x = (a, b), the quadratic value is a² − 2ab + b² = (a − b)². It is never negative, so Q is psd.
However, x = (1, 1) is nonzero and gives zero. Q is therefore not positive definite. Its eigenvalues are 0 and 2, with directions (1, 1) and (1, −1), respectively. Its determinant is zero and it has no ordinary inverse.
Consider the fitting objective F(a, b) = (a − b − 2)²/2. Its Hessian is Q. Every point with a − b = 2 minimizes F, giving objective zero. Examples include (2, 0), (1, −1), and (5, 3). The model identifies the difference but not the shared offset.
Add the penalty (a² + b²)/2. The new Hessian is Q + I, whose eigenvalues are 1 and 3. Solving the new first-order equations gives 2a − b = 2 and −a + 2b = −2, hence a = 2/3 and b = −2/3.
The unique penalized solution has difference 4/3, so regularization does more than choose among the former exact fits: it also shrinks the fitted difference. Its data loss is 2/9 and its penalty is 4/9, totaling 2/3. These calculations are hand-checked illustrations, not a software trace.
Illustrative case study
Fictional case: Brook Networks smooths readings over a connected device graph. Its team expects the graph penalty alone to select one complete vector of values. Instead, adding the same constant to every fitted reading leaves that penalty unchanged.
Analysts recognize the constant-vector nullspace and introduce an appropriate data-fitting term or an explicit reference condition, depending on the intended task. They avoid treating singularity as evidence that the graph is necessarily corrupt. The zero direction has a meaningful interpretation: the penalty measures disagreement, not the absolute level. No deployment outcome is claimed.
Common misconceptions and mistakes
Positive entries do not prove positive semidefiniteness. A matrix can have positive entries and still produce a negative quadratic value in some direction. Negative off-diagonal entries likewise do not disprove it, as the worked example shows.
Do not turn the square root of xᵀQx into a claimed norm without checking definiteness. For singular psd Q it is generally a seminorm, since a nonzero vector can have value zero.
A psd Hessian also does not force a minimum for every quadratic objective. The function −a, viewed as a quadratic with zero Hessian, is unbounded below over unrestricted real a.
Business use
The practical value is knowing which variations the model can distinguish. Inspect zero or near-zero directions before attempting inversion or interpreting parameter uncertainty. Redundant features, graph components, or insufficient observations can explain them.
A diagonal adjustment can make a psd matrix positive definite, but it changes the mathematical problem. Record the adjustment and check whether it is intended as statistical regularization, numerical stabilization, or both. These purposes can require different choices.
People also ask
Is the zero matrix psd?
Yes. Every quadratic value is zero.
Can a psd matrix be invertible?
Yes. A symmetric psd matrix is invertible exactly when all its eigenvalues are positive.
Does psd mean entrywise nonnegative?
No. It means nonnegative quadratic forms in every direction.
Check your understanding
Why is BᵀB psd even when B contains negative entries?
Answer: For every x, xᵀBᵀBx = (Bx)ᵀ(Bx), a sum of squares, which cannot be negative.
Related terms
- Positive definite (pd)
- Gram matrix
- Covariance matrix
- Nullspace
- positive definite
- graph Laplacian
- convexity
- seminorm
Learn next
connect nullspaces with nonunique parameter estimates, then examine how a regularizer changes both the spectrum and the fitted solution.
More in Mathematics and optimization
- Absolute error loss - Absolute error loss measures a numerical prediction's error as the absolute difference between its predicted and reference values: L(y,y-hat)=|y-y-hat|.
- Active-set method - An active-set method is an optimization approach that works with a changing subset of constraints believed to be tight at the solution.
- Adjoint state method - The adjoint state method computes how a chosen objective changes when the parameters of a mathematical model change.
- Σ-algebra
- Algebraic connectivity - Algebraic connectivity is the second-smallest eigenvalue of a graph's Laplacian matrix.
- Algorithm IMED - Algorithm IMED, short for Indexed Minimum Empirical Divergence, is a method for choosing actions in a stochastic multi-armed bandit problem.
Sources
Primary source: Aalto Dictionary of Machine Learning, positive semi-definite, positive definite, and nullspace entries: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . Relevant definitions were read. The two-variable example, regularization calculation, and fictional graph case are original. Matrix ordering is explicitly distinguished from entrywise comparison.
Previous: Positive definite (pd) Next: Positively skewed