AI Glossary / AI and machine learning / Advanced

Product of experts

Definition

A product of experts is a probabilistic model that combines several component models by multiplying their scores for the same possible outcome and then normalizing the result. Each component can express a different constraint. Outcomes that receive support across the components tend to survive the combination.

What it means

The word expert means a mathematical component, not a person or a guarantee of specialist knowledge. The components assess a shared outcome space, such as possible images or class labels. A component can strongly reduce support for an outcome even when another component likes it. For a business reader, the useful distinction is between combining requirements and taking an average opinion. A product can express the former, but whether those requirements are reliable remains an empirical question.

How it works

For each candidate outcome, multiply the nonnegative expert scores. Divide this product by the sum of such products across all possible discrete outcomes; continuous models require the corresponding integral. That normalizing quantity makes the combined values a probability distribution. It can also be difficult to compute in large models. Learning therefore depends on the model family and available approximations. Hinton's research develops contrastive divergence as an approach to training such models. Implementation must handle very small numbers, commonly through logarithms, and evaluate the resulting model against held-out observations.

Three practical examples

• A hypothetical document classifier combines signals for vocabulary and page layout. A candidate document type needs reasonable support from both, rather than receiving a high score from vocabulary alone.
• An image research model combines constraints on broad shape and local visual patterns. An image with the right outline but implausible details can receive a low combined score.
• A planning prototype scores proposed configurations against separate learned compatibility preferences. The team checks that legitimate unusual configurations are not rejected simply because one component has seen few examples.

Think of it this way

Think of several transparent stencils placed over the same page. The overlap leaves fewer visible regions than any single stencil. The analogy captures combined constraints, but probabilities are usually graded rather than simple holes or solid areas. A poorly designed stencil can hide the answer everyone actually wanted.

Worked example

Suppose three outcomes, A, B and C, receive probabilities 0.6, 0.3 and 0.1 from one expert, and 0.2, 0.5 and 0.3 from another. Their products are 0.12, 0.15 and 0.03. These sum to 0.30, so the normalized combined probabilities are 0.40, 0.50 and 0.10. B becomes the most likely outcome. An equal arithmetic mixture would instead give 0.40, 0.40 and 0.20. These invented values show how multiplication and averaging differ; they do not establish which combination will make better predictions on real data.

Illustrative case study

A fictional records team tests a product model for classifying incoming business forms. One component captures text patterns, while another captures spatial organization. The team reserves examples from previously unseen suppliers for evaluation and compares the product with a simple averaging baseline. It discovers that a new supplier's valid form has an unfamiliar layout, causing the layout component to suppress the correct answer. The team introduces an exception review process and investigates component reliability before expansion. Success means better measured classification and manageable review effort, not merely a sharper probability distribution.

Common misconceptions and mistakes

A product of experts is not a mixture of experts: multiplying component scores has different behavior from adding weighted outputs or selecting a component. More components do not automatically improve accuracy. Repeated evidence can produce excessive confidence, especially when components make similar errors. Normalization cannot repair incorrect scores or missing support. If every candidate has a zero product, the proposed distribution is undefined and the model needs attention. A confident combined output should not be interpreted as proof that its contributing components are independent or correct.

Business use

The concept helps teams evaluate systems that combine multiple compatibility signals or constraints. Ask what each component contributes, how failures are detected, and whether performance improves over simpler combinations. Include evaluation of probability calibration and unusual cases alongside accuracy. Budget for training and normalization costs appropriate to the model. Where decisions affect customers or operations, preserve a route for reviewing cases that one unreliable component effectively vetoes. The right comparison is the complete operating process, including exceptions and maintenance.

People also ask

Must an expert be a neural network?

No. The combination describes a mathematical structure, and components can take different forms.

Can one component suppress an outcome?

Yes. A very small score reduces the product, although the final probability also depends on competing outcomes.

Is a sharper distribution always better?

No. Concentrated probability is useful only when it corresponds to reliable predictions and appropriate uncertainty.

Check your understanding

If two components assign an outcome scores 0.8 and 0.1, is its combined probability necessarily 0.08?

Answer: No. That is its unnormalized product; other outcomes determine the normalizing denominator.

Why test rare but legitimate inputs?

Answer: A component with poor coverage can suppress them even when other evidence is strong. Explain why averaging and multiplication need separate evaluation rather than interchangeable labels.

Related terms

• Mixture of experts
• Ensemble learning
• Energy-based model
• Contrastive divergence
• Probability calibration
• partition function

Learn next

Compare a product and a mixture using a small probability table. Then study partition functions and energy-based models to understand why training becomes harder when the outcome space is large. Finally, examine calibration checks for models that combine correlated signals.

More in AI and machine learning

• Probabilistic numerics - Probabilistic numerics treats numerical computation as inference about an unknown mathematical quantity.
• Probability matching - Probability matching is a choice pattern in which alternatives are selected in proportions resembling their perceived probabilities or observed frequencies.
• Procedural memory - Procedural memory, in an AI agent, is the stored knowledge of how to do things: reusable routines, skills and rules for carrying out tasks, as distinct from...
• Programming by example - Programming by example is an approach to creating programs in which a user supplies examples of the desired behavior and a system infers a program...
• Proper generalized decomposition - Proper generalized decomposition, or PGD, is a numerical modeling approach that approximates a solution as a sum of products of simpler functions.
• Prototype methods - Prototype methods are machine-learning approaches that represent categories or patterns through selected or learned representative examples, called prototypes.

Sources

Geoffrey Hinton, Training Products of Experts by Minimizing Contrastive Divergence: https://www.cs.toronto.edu/~hinton/absps/tr00-004.pdf Hinton's publications on products of experts: https://www.cs.toronto.edu/~hinton/poe.html The numerical example and business scenarios above are original illustrations, not reported deployments or performance results. Independent review consulted Hinton's original technical report, especially the normalized product formula, mixture comparison and contrastive-divergence discussion. Both the normalized product and arithmetic-mixture examples were recomputed independently.

Previous: Proctor survey Next: Product rule