Must all training records contain every modality?

Some architectures and objectives support partially observed records, but cross-modal relationships still require adequate training evidence.

Is a multimodal VAE the same as a conditional VAE?

They can overlap, but a conditional model for one output given another does not necessarily model a joint distribution with flexible inference in both directions.

Are latent variables directly interpretable attributes?

Not necessarily; interpretability needs separate evidence.

AI Glossary / Probabilistic machine learning / Advanced

Multimodal variational autoencoder

Definition

A multimodal variational autoencoder is a generative model that learns latent representations from more than one type of data, such as images, text, or sensor readings. It combines modality-specific observation models with approximate inference, supporting joint generation and, when appropriately designed and trained, inference from incomplete combinations of modalities.

What it means

Different data types can describe the same underlying situation. An image may show an object, while text describes it and a sensor measures one of its properties. A multimodal VAE attempts to learn relationships among these observations through latent variables.
The model does more than concatenate features for a classifier: it specifies how observations can be generated from a latent representation. It can use available observations to infer that representation and generate a possible missing modality. Generated content is a model prediction, however, not a recovered observation known to be true.

How it works

A common construction uses a shared latent variable z, a prior over z, and separate decoders for the modalities. Given z, the observation models may assume conditional independence. Encoders approximate the latent posterior from whichever inputs are available.
In the product-of-experts MVAE of Wu and Goodman, modality-specific encoder factors combine with one prior factor. For Gaussian factors, the combined precision is the sum of their precisions, making the calculation tractable. A missing modality contributes no encoder factor. This is one architecture within the broader family; not every multimodal VAE uses a product of experts.
Training balances reconstruction terms with a divergence from the prior. Training on appropriate combinations of complete and partial inputs helps the inference system handle missing modalities. The ability to omit an input computationally does not guarantee that the model learned a reliable relationship between modalities.

Three practical examples

1. A product-data team could study joint representations of photographs and descriptions, then evaluate proposed descriptions when only an image is available.
2. An equipment researcher could combine sound recordings and vibration measurements, examining how the representation behaves when one sensor stream is unavailable.
3. An archive researcher could model related images and structured attributes, exploring both joint generation and prediction of attributes from images.
Benefits depend on alignment, data quality, task relevance, and training.

Think of it this way

Imagine several witnesses describing the same scene in different ways: one supplies a sketch, another a written account, and another measurements. A shared representation tries to reconcile their accounts. If a witness is absent, the others can still suggest what happened. A confident reconciliation can nevertheless be wrong, especially when an account is misleading or the shared assumptions are inappropriate.

Worked example

Consider an original one-dimensional Gaussian product-of-experts illustration. Let the prior factor be Normal(0, 1). Suppose an image encoder supplies a Gaussian expert factor with mean 2 and variance 1/4, and a text encoder supplies another with mean −1 and variance 1.
These encoder outputs are stipulated expert factors, not already-complete single-modality posteriors that contain the prior. We form the approximate joint distribution proportional to the prior times both expert factors, then normalize it.
The three precisions, meaning inverse variances, are 1, 4, and 1. Their sum is 6, so the combined variance is 1/6. The precision-weighted mean is (1 × 0 + 4 × 2 + 1 × (−1)) / 6 = 7/6, approximately 1.166667.
If text is unavailable, use only the prior and image factor. Their total precision is 5, variance is 1/5, and mean is 8/5, or 1.6. If only text is available, the corresponding mean is −1/2 and variance is 1/2.
Adding the text factor moves the combined mean and narrows the Gaussian distribution, even though its preferred value differs from the image factor's. Narrower variance is a consequence of this chosen product, not proof that the sources agree or that the estimate is calibrated.
The example demonstrates latent fusion only. No encoder was trained, no decoder output was generated, and no missing information was verified. Multiplying complete single-modality posteriors and adding the prior again would be a different calculation that can count prior information repeatedly.

Illustrative case study

In a fictional scenario, Hazel Catalog studies paired photographs and text descriptions of manufactured items. Some records contain both modalities; others contain only one. The team evaluates a multimodal VAE trained with complete records and suitable partial-input combinations.
Researchers test image-only, text-only, and joint inference separately. They inspect whether generated descriptions remain consistent with available evidence and whether one modality dominates the shared representation. They also compare against simpler models for the actual catalog task.
The fictional lesson is to evaluate missing-input behavior directly and label generated content as generated. No reliable recovery of absent descriptions or improvement in catalog quality is claimed.

Common misconceptions and mistakes

Multimodal does not mean that the model must work equally well with every input combination. A shared latent variable does not establish a causal explanation. Conditional independence given z is a modeling assumption, not a statement that the observed modalities are unrelated. Gaussian product fusion can become confident even when expert factors disagree. Entirely unpaired collections do not automatically identify the desired correspondence between modalities. A plausible generated image or sentence is not evidence that it was present in the original record.

Business use

This model family can be useful when linked data types are available and incomplete records are common. Begin with the intended decision or generation task, then test each input-availability pattern that matters operationally. Review whether training pairs refer to the same entity or event, whether missingness is informative, and whether one modality overwhelms another during training. Keep generated replacements distinguishable from recorded data, especially if later systems will learn from the resulting records.

Check your understanding

Why is the combined variance 1/6 in the example?

Answer: The prior and two expert precisions sum to six.

What changes when text is missing?

Answer: Its expert factor is omitted.

Why does a smaller variance not establish correctness?

Answer: The product reflects the assumed factors and their confidence, which may be inaccurate or inconsistent with real data.

Related terms

Variational autoencoder
Multimodal Learning
Product of experts
Latent variable
evidence lower bound
missing modalities
cross-modal generation

Learn next

Compare product-of-experts and mixture-of-experts inference, then examine shared versus modality-specific latent representations and missing-input evaluation.

Sources

Wu and Goodman, Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018), Section 2: https://papers.neurips.cc/paper/7801-multimodal-generative-models-for-scalable-weakly-supervised-learning.pdf . The Gaussian-factor example and Hazel Catalog scenario are original illustrations. Section 2 independently read; Gaussian arithmetic checked.

Previous: Multimodal Transformer Next: Multinomial distribution