Is it still a Gaussian mixture?

Yes, after factors are integrated out, each component is Gaussian with a factor-structured covariance.

How does it differ from a mixture of probabilistic PCA models?

Factor analysis allows diagonal noise with potentially different variances by observed dimension; probabilistic PCA usually imposes isotropic noise within a component.

Must all components have different means?

No. The worked example uses equal means and distinguishes components through their covariance structure.

AI Glossary / Probabilistic machine learning / Advanced

Mixture of factor analyzers

Definition

A mixture of factor analyzers is a probabilistic model that combines several factor-analysis components. Each component describes observations using a small set of continuous latent factors plus noise, while a discrete component variable allows different local patterns of variation.

What it means

One global factor model assumes that the same underlying directions explain variation throughout the dataset. That can be restrictive when different groups have different relationships among features. A mixture allows those relationships to vary across components.

The model combines soft clustering with local dimensionality reduction. An observation can have partial membership probabilities across several components, and each component supplies its own low-dimensional explanation.

After the continuous factors are integrated out, the observed distribution is a Gaussian mixture with structured component covariances. The factor structure reduces flexibility relative to unrestricted covariances, which can be useful when observations are high-dimensional and data are limited.

How it works

First draw a component label k using mixing probabilities. Draw a low-dimensional standard Gaussian factor vector z, then generate x = mu_k + W_k z + noise. The mean mu_k locates the component and the loading matrix W_k maps factors into observed coordinates.

With diagonal noise covariance Psi_k, the component covariance is W_k W_k transpose + Psi_k. The diagonal noise describes residual variation independent across observed coordinates conditional on the factors and component. Some formulations share the noise covariance across components; others allow it to differ.

Expectation maximization can alternate between estimating component responsibilities and factor moments, then updating parameters. Component count and factor dimension require selection. Multiple starts, regularization and validation help assess local optima and overfitting; an exact EM update does not promise a globally optimal fitted model.

Three practical examples

A sensor-analysis project models different operating regimes, each with a small number of correlated variation patterns.
An image-research team explores local linear representations for different families of image appearances.
A product-usage analyst tests whether separate groups need different relationships among measured features rather than one common factor loading matrix.

Think of it this way

Think of approximating a curved landscape with several local flat patches. Each patch captures a different direction of variation, and their combination can describe more structure than one flat sheet.

The analogy explains local linear modeling. Mixture components have probability densities and overlapping responsibilities; they are not necessarily nonoverlapping geographic patches or true natural categories.

Worked example

Two components with the same center but different directions

Consider two equally weighted components in two observed dimensions. Both have zero mean and one standard Gaussian factor. Component A has loading vector (2, 0); component B has loading vector (0, 2). Both use identity noise covariance.

Their observed covariances are therefore diagonal matrices with entries (5, 1) for A and (1, 5) for B. Both determinants equal 5, so their Gaussian normalizing factors match.

For observation x = (2, 0), A's squared Mahalanobis distance is 4/5, while B's is 4. The density ratio A to B is exp((4 − 0.8)/2) = exp(1.6), approximately 4.9530.

With equal prior component weights, A's responsibility is exp(1.6)/(1 + exp(1.6)), approximately 0.832018. This is a soft assignment: B still receives about 0.167982 despite the observation lying along A's main direction.

Within A, the factor posterior has variance 1/(1 + 4) = 0.2 and mean 0.2 times 2 times 2 = 0.8. The factor estimate is not simply the observed first coordinate divided by two, because the model accounts for observation noise.

Identical component means do not make the components identical. Their covariance directions distinguish them, while the posterior retains uncertainty about both component membership and factor value.

Illustrative case study

A fictional industrial-monitoring team finds that several sensor readings move together during one operating mode but follow different relationships during another. A single global factor model leaves structured residuals.

The team compares a mixture of factor analyzers with a single factor model and simpler Gaussian mixtures. It evaluates complete held-out runs, inspects residual noise assumptions and repeats fitting from different starting configurations.

The analysts use component probabilities as modeling summaries, then compare them with recorded operating modes. They do not automatically name every component a physical regime or interpret a latent factor as a causal mechanism. This case makes no claim of measured detection improvement.

Common misconceptions and mistakes

It is PCA followed by clustering: the probabilistic mixture fits membership and local factor structure jointly.
Low-dimensional factors make the observed covariance singular: positive residual noise can make the full covariance nonsingular.
Each observation must belong with certainty to one component: posterior responsibilities are generally fractional.
Factor names are automatically identifiable: rotations of a factor representation can preserve its implied covariance, and mixture component labels can be permuted.

Business use

The model can offer a middle ground between a restrictive global representation and a Gaussian mixture with expensive unrestricted covariance matrices. It is useful to investigate when groups exhibit different feature dependencies.

Choose component and factor counts using a defensible validation procedure, and report uncertainty in memberships when assignments overlap. Check sensitivity to feature scaling, residual-noise restrictions and small components. Constraints on noise variance may be needed to prevent unstable fits. A compact density representation does not by itself establish actionable customer segments or reliable anomaly labels.

Check your understanding

A component has loading vector (1, 2) and identity noise covariance. What is its observed covariance?

Answer: The loading outer product has rows (1, 2) and (2, 4). Adding the identity gives rows (2, 2) and (2, 5). The shared factor creates the off-diagonal covariance.

Related terms

Factor analysis
Gaussian mixture model
Latent variable
Expectation-maximization
loading matrix
responsibility
probabilistic PCA
local dimensionality reduction

Learn next

Derive the covariance of one factor analyzer, then combine two components with different loading directions. Calculate membership probabilities for a few points before fitting a larger model. Compare shared and component-specific diagonal noise assumptions and examine their practical consequences.

Sources

Ghahramani and Hinton, The EM Algorithm for Mixtures of Factor Analyzers, technical report, 1996, revised 1997: https://www.cs.utoronto.ca/~hinton/absps/tr-96-1.pdf

Tang, Salakhutdinov and Hinton, Deep Mixtures of Factor Analysers, background formulation: https://www.cs.toronto.edu/~hinton/absps/deep_mfa.pdf

The two-component calculation and fictional monitoring case are original teaching illustrations. Shared and component-specific noise formulations are distinguished explicitly.

Previous: Mixture of experts Next: Mixture of Gaussians