Is this related to linear discriminant analysis?

Yes. Gaussian class models with a common covariance lead to the usual linear discriminant scores.

Can the covariance contain correlations?

Yes; tied means shared across classes, not diagonal.

Must every class have the same number of observations?

No. Pooling can use all within-class residuals with the chosen estimation convention.

AI Glossary / Probabilistic machine learning / Advanced

Tied covariance in Gaussian discriminant analysis

Definition

Tied covariance in Gaussian discriminant analysis means that all modeled classes share one covariance matrix, while retaining their own mean vectors and class probabilities. This constrains the classes to have the same within-class spread and correlation structure. With a nonsingular shared covariance, the resulting pairwise classification boundaries are linear in the original features.

What it means

Gaussian discriminant analysis describes the distribution of features within each class, then combines those distributions with class probabilities to predict a label. Tying covariance shares information about variation across classes rather than estimating a separate matrix for each one.

The assumption concerns variation around each class's own mean. It does not say that the class means are equal, that every feature has the same variance, or that features are independent. A shared full covariance can contain unequal diagonal entries and nonzero correlations. The tradeoff is fewer covariance parameters in exchange for a stronger assumption about class geometry.

How it works

Estimate a mean vector for each class and subtract the appropriate class mean from every training observation. Pool the outer products of these within-class residuals to estimate the common covariance. Under the standard maximum-likelihood convention, divide their sum by the total number of observations. An unbiased pooled estimator uses a different denominator; implementations should state their convention.

For class k, compute a discriminant score consisting of x-transpose Sigma-inverse mu-k, minus one half mu-k-transpose Sigma-inverse mu-k, plus log of the class prior. Choose the largest score for equal misclassification costs.

The shared quadratic term in x cancels when comparing classes, producing linear boundaries. Separate covariance matrices generally retain different quadratic terms, leading to quadratic discriminant analysis.

If the shared covariance is singular or poorly conditioned, direct inversion is unreliable. Regularization, shrinkage or an appropriate reduced representation may be necessary. Those choices alter the estimator and should be selected using training and validation data rather than final evaluation outcomes.

Three practical examples

1. A quality-control team classifies measurements from two product conditions while assuming comparable within-condition measurement variation.

2. A document-analysis team uses a regularized shared covariance with continuous feature vectors when separate class estimates would be too noisy.

3. A teaching comparison fits linear and quadratic discriminant models to show how sharing covariance changes both parameter count and decision-boundary shape.

Think of it this way

Think of placing identical elliptical templates at different centers on a map. Each class gets its own center, but the ellipse's orientation and shape are shared. Different class probabilities can still influence which template wins at a location. Sharing the template does not merge the centers or make the map's horizontal and vertical variation identical.

Worked example

Consider a one-feature model with class A mean zero, class B mean four, and shared variance four. Initially give both classes prior probability one half. These are stipulated model parameters, so no covariance estimation is needed for this calculation.

The log posterior odds of B against A are [minus (x minus 4) squared plus x squared] divided by eight. Expanding gives x minus two. Thus the equal-cost decision boundary is x = 2, and a value x = 3 gives log odds one.

The posterior probability of B at x = 3 is 1/[1 + exp(minus one)], approximately 0.7311. Sharing the variance has produced a linear score even though each class density is Gaussian.

Now change the priors to P(A) = 0.75 and P(B) = 0.25 while keeping means and variance fixed. The log odds become x minus two plus log(1/3). The boundary moves to x = 2 + log(3), approximately 3.0986.

The same covariance assumption therefore does not imply that the midpoint is always the decision boundary. Priors and decision costs matter. These posterior values also depend on the model assumptions; they are not evidence that the real-world probabilities are calibrated.

Illustrative case study

Fictional case: Hazel Instruments classifies two operating conditions from a modest collection of sensor readings. Separate covariance estimates vary sharply between resampled training sets, so analysts compare them with a regularized tied estimate.

The shared model gives more stable validation predictions, but the team finds one feature with substantially different class spreads. It investigates whether the simpler assumption remains acceptable for the intended decision and checks performance across operating ranges. The final choice rests on independent predictive evidence and error costs, rather than treating fewer parameters as automatically better.

Common misconceptions and mistakes

Tied covariance is not the covariance of all observations around one overall mean. That overall covariance includes between-class separation, while the tied estimate pools residuals around class-specific means.

It is also not a diagonal-covariance assumption. Correlations can be shared across classes without being zero. Gaussian naive Bayes makes a different structural restriction.

Linear boundaries do not mean the class densities are linear functions. Nor does sharing covariance guarantee a stable estimate when the number of features is large compared with the available observations. Regularization and careful feature handling may still be essential.

Business use

For business classification, tied covariance offers an interpretable baseline and can reduce estimation demands when individual classes have limited data. Compare it with alternative models using suitable validation splits, calibration checks and decision costs. Record class priors separately from covariance assumptions, especially when deployment class frequencies differ from the training sample. A convenient model should remain accountable to measured performance.

Check your understanding

Why do quadratic terms cancel?

Answer: Every class comparison contains the same x-transpose Sigma-inverse x term.

Would changing class priors change the common covariance?

Answer: Not inherently; priors are separate model parameters, although a full refit may change several estimates.

Related terms

Gaussian discriminant analysis
Linear discriminant analysis
quadratic discriminant analysis
pooled covariance
covariance shrinkage
class prior and Mahalanobis distance

Learn next

Next, derive the two-class discriminant score from Gaussian log densities. Then compare pooled within-class covariance with total covariance and investigate how shrinkage affects an ill-conditioned estimate.

Sources

Primary source: Andrew Ng, Stanford CS229 notes, Generative Learning Algorithms, Gaussian Discriminant Analysis model and maximum-likelihood estimates: https://cs229.stanford.edu/notes2020spring/cs229-notes2.pdf

The numerical classification example and fictional case are original teaching material. The worked example uses specified parameters and equal decision costs unless explicitly changed.

Independent review consulted Sections 1.2–1.3 of the official CS229 notes, including the pooled maximum-likelihood covariance.

Previous: Thurstone model Next: Tiered indexes