Can the covariance be singular?

Yes, under the definition used here.

AI Glossary / Statistics and probability / Advanced

Multivariate normal distribution

Definition

The multivariate normal distribution is a probability distribution for a vector whose every linear combination is normally distributed, allowing constant combinations with zero variance. It is also called the multivariate Gaussian distribution.

What it means

A mean vector describes the center, and a covariance matrix describes spread and linear dependence. Having normal distributions for individual coordinates is not sufficient to establish joint normality. Within a jointly normal vector, uncorrelated coordinates are independent; that implication does not hold for arbitrary distributions.

How it works

Start with independent standard normal variables in a vector Z and transform them as X=mu+AZ. The mean is mu and the covariance is AA-transpose. A positive-definite covariance gives a density over the full space. A singular covariance can still define a normal distribution, but its outcomes lie in a lower-dimensional subspace.
Linear transformations preserve joint normality. These properties make the model useful for calculations involving combinations and conditional distributions, provided the assumed joint model is appropriate.

Three practical examples

A fictional calibration team models two simultaneous measurement errors and checks the distribution of their difference.
A fictional simulation exercise generates correlated Gaussian features to test whether an analysis recovers the specified covariance.
A fictional operations team compares a joint Gaussian approximation with observed paired forecast errors before using it to estimate combined uncertainty.

Think of it this way

Imagine a circular cloud of points that is stretched, tilted, and shifted. The transformation creates an elliptical cloud while retaining its Gaussian structure. Flattening the cloud completely onto a line creates a singular case rather than an ordinary two-dimensional density.

Worked example

For an original two-variable model, draw independent standard normal variables Z1 and Z2. Define X=20+2Z1 and Y=30+Z1+sqrt(8)Z2.
The means are 20 and 30. The variance of X is four. The variance of Y is one plus eight, or nine. Their covariance is two because the only shared random contribution comes from Z1. Their correlation is therefore 2/(2 times 3)=1/3.
The covariance matrix has rows (4, 2) and (2, 9). Its determinant is 36 minus four, or 32, which is positive. Together with the positive first diagonal entry, this verifies positive definiteness in this two-dimensional example.
Suppose an illustrative simulation draw has Z1=1 and Z2=0. The transformed values are X=22 and Y=31. This is a calculation for a chosen draw, not a claim that those exact values have positive probability under a continuous model.
Now consider the total S=X+Y. Its mean is 50 and its variance is four plus nine plus twice two, or 17. Its standard deviation is approximately 4.123. Because the total is normal, the probability that it exceeds 50+sqrt(17), approximately 54.123, is about 0.15866.
The difference D=X-Y has mean minus ten and variance four plus nine minus twice two, or nine. Its standard deviation is three. Positive covariance increases the variance of this sum while reducing the variance of this difference.
Next consider the conditional distribution of Y when X=24. The construction tells us that Z1=(24-20)/2=2. The remaining independent contribution is sqrt(8)Z2, so the conditional distribution of Y is normal with mean 32 and variance eight.
Thus observing X changes the predicted mean of Y and reduces its variance from nine to eight in this model. Conditioning on an exact continuous value is understood through the conditional distribution, not as dividing by the probability of a positive-mass event at X=24.
To examine joint density, compare two points: A=(22, 33) and B=(22, 27). Each coordinate is one marginal standard deviation from its mean, but A's deviations have matching signs and B's have opposite signs.
The inverse covariance matrix is one thirty-second times the matrix with rows (9, -2) and (-2, 4). Applying this matrix to the deviations gives squared Mahalanobis distance 1.5 for A and three for B.
The Gaussian density is proportional to exp(-q/2), where q is that squared distance. The density ratio at A versus B is therefore exp((3-1.5)/2)=exp(0.75), approximately 2.117. Marginal distances alone miss the effect of the positive relationship between coordinates.
For this two-dimensional model, q has the same distribution as Z1 squared plus Z2 squared. The probability inside q no greater than four is 1-exp(-2), approximately 86.47 percent. Inside q no greater than one it is 1-exp(-0.5), approximately 39.35 percent. A familiar one-dimensional coverage percentage cannot simply be assigned to a joint ellipse.
For a separate singular example, keep X=20+2Z1 but define Y=30+3Z1, removing Z2. The covariance matrix becomes rows (4, 6) and (6, 9), with determinant zero. Every point satisfies Y-30=1.5(X-20).
This remains jointly normal under the definition allowing degeneracy, but there is no ordinary density with respect to area across the plane. Once X is known, Y is determined exactly. Adding an arbitrary small variance would change this model, not merely rewrite it.

Illustrative case study

Fictional company Cedar Calibration uses the nonsingular construction to explain correlated measurement errors. Reviewers compare component variation, total variation, and the conditional prediction.
Before applying the model to actual measurements, the team examines the joint observations and checks whether the Gaussian assumptions describe the intended operating conditions. It documents any transformations and limitations. This fictional case illustrates how a convenient probability model should remain connected to empirical evidence.

Common misconceptions and mistakes

Normal-looking marginal distributions do not prove joint normality.
Zero covariance implies independence here because of joint normality.
An ellipse's coverage is not a one-dimensional interval's coverage.
A singular covariance does not invalidate the distribution, but it changes the appropriate density description.

Business use

Use the model when its joint assumptions adequately support the required calculation. In the fictional calibration setting, a decision about a difference needs the covariance as well as each measurement's variance. Check that predicted combinations match observed behavior.

Check your understanding

What is the example's correlation?

Answer: One third.

What is the variance of X+Y?

Answer: Seventeen.

What is Y's conditional mean when X=24?

Answer: Thirty-two.

Related terms

Covariance matrix
Mahalanobis distance
Conditional distribution
Linear transformation
Gaussian random vector
joint normality
singular distribution

Learn next

Study conditional Gaussian calculations and compare Gaussian models with distributions that share their covariance but have different joint behavior.

Conditional Gaussian

Sources

Aalto Dictionary of Machine Learning, Gaussian random variable: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . Pishro-Nik, Bivariate Normal Distribution: https://www.probabilitycourse.com/chapter5/5_3_2_bivariate_normal_dist.php . Numerical examples and company scenario are original.

Previous: Multivariate model Next: Multivariate outliers