Are all Boltzmann machines deep?

No. Depth depends on the architecture; the general concept does not require many hidden layers.

Does lower energy mean higher probability?

At a fixed positive temperature within the same normalized model, yes. Comparing raw energies across unrelated models is a different matter.

Can hidden units have connections to one another?

A general Boltzmann machine can allow them. A restricted Boltzmann machine removes within-layer connections as part of its defining structure.

AI Glossary / Deep learning / Advanced

Boltzmann Machine Network

Definition

A Boltzmann machine network is a stochastic neural model that assigns probabilities to configurations through an energy function. Lower-energy configurations receive greater probability under its Boltzmann distribution. Learning adjusts connections and biases so the model's distribution better matches observed data. Visible units represent observations, while hidden units can represent unobserved structure. The term includes a general model family; restricted and deep Boltzmann machines impose particular connection patterns.

What it means

Unlike a basic feedforward predictor that calculates one output by passing values through a fixed sequence, a Boltzmann machine describes a distribution over possible joint states. Sampling and inference explore states under that distribution.

“Energy” is a mathematical score derived from unit states and parameters, not necessarily the electricity consumed by a computer. “Temperature” controls the distribution's sensitivity to energy differences in formulations that expose it. The model's connection to statistical physics supplies useful mathematics, but it does not mean every use requires physical simulation hardware or a quantum computer.

How it works

In a classic binary formulation, symmetric connection weights and biases define the energy of each configuration. The probability is proportional to the exponential of negative energy, with a normalization term summing over all configurations. Hidden states are summed out when calculating probabilities of visible observations.

Learning compares statistics associated with observed data against statistics generated by the model. The data-associated phase encourages compatible configurations; the model-associated phase counterbalances configurations the current model produces too readily.

Exact sums and equilibrium sampling become difficult as the network grows. Practical methods use sampling or other approximations. A restricted Boltzmann machine simplifies connectivity to make parts of inference and learning easier, but that restriction is not present in every Boltzmann machine.

Three practical examples

1. Distribution-learning demonstration: students train a tiny model on simple binary patterns and inspect whether its generated pattern frequencies resemble the training distribution.

2. Missing-component experiment: researchers hold some visible values fixed and investigate the model's conditional distribution over the unknown components rather than demanding a single certain completion.

3. Energy-model comparison: a team compares general and restricted architectures on a small research task, measuring sampling cost and approximation quality alongside predictive or generative performance.

Think of it this way

Think of a landscape in which valleys represent more favored configurations, while random motion sometimes visits higher ground. Learning reshapes the landscape so its pattern of visits better reflects the data. The analogy is limited: a neural configuration space is not an ordinary physical landscape, and exploring it adequately can be difficult. Seeing one appealing sample does not show that the entire distribution has been learned well.

Worked example

Consider a two-unit binary model with states 0 or 1, zero biases, temperature one, and energy E(v,h) = −wvh. Choose w = ln(3).

The states (0,0), (0,1), and (1,0) each have energy zero and unnormalized probability weight 1. State (1,1) has energy −ln(3), giving weight exp(ln(3)) = 3. The normalization constant is therefore 1 + 1 + 1 + 3 = 6.

The joint probability of (1,1) is 3/6 = 0.5; each other state has probability 1/6. If v is fixed at 1, the two compatible states have relative weights 1 and 3, so P(h = 1 | v = 1) = 3/4.

This exactly enumerable example illustrates joint and conditional probabilities. It is not a trained network or a demonstration that a larger model can calculate its normalization constant efficiently.

Illustrative case study

Fictional case: Energy Cove, a research team, presents generated binary patterns from a small Boltzmann model. Selected samples look like the patterns in its training collection. A reviewer asks whether their frequencies also match, and whether important alternatives are missing.

The team compares generated statistics across repeated sampling runs and investigates sensitivity to initialization and sampling duration. It discovers that visual plausibility alone is an incomplete evaluation of a probability model. No numerical performance result is claimed. The example highlights the difference between producing some familiar-looking configurations and adequately representing the distribution of configurations relevant to the task.

Common misconceptions and mistakes

A Boltzmann machine is not simply a deterministic Hopfield network with a different name. Its stochastic states and probabilistic learning interpretation are central distinctions.

The negative learning phase is not necessarily a dataset of examples labeled “wrong.” It concerns statistics under the model's own distribution.

Low energy does not mean factual truth or safety; it means relative preference under the specified model. Nor does one short sampling run guarantee equilibrium. Finally, a restricted Boltzmann machine and a deep Boltzmann machine are particular architectures. Results or update rules for one should not be transferred to the general model without checking the assumptions.

Business use

Boltzmann machines are useful foundations for understanding energy-based generative modeling and specialist probabilistic research. For an operational proposal, a business should ask what advantage the model offers for its actual data and decision process.

Evaluate the full cost of learning and inference, including repeated sampling and uncertainty in approximate calculations. Compare appropriate alternatives using the same data and task criteria. If generated samples are used for another workflow, inspect diversity, frequency, and failure cases rather than judging a few examples. Document the architecture, energy convention, sampling method, and stopping choices so another team can reproduce the claimed behavior instead of relying on an attractive animation of changing neural states.

Check your understanding

In the toy model, state (1,1) has unnormalized weight 3. Is its probability 3?

Answer: No. Divide by the sum of all four weights, 6. Its joint probability is 0.5. Unnormalized weights are not themselves probabilities.

Related terms

Energy-based model
Hidden units
Gibbs sampling
Markov chain Monte Carlo
Restricted Boltzmann Machine
Boltzmann distribution
partition function
hidden unit
deep Boltzmann machine

Learn next

Enumerate every state of a two- or three-unit model and calculate its exact probabilities. Then compare those probabilities with frequencies from a documented sampler before studying approximate learning in larger networks.

Sources

Primary researcher teaching sources: Hinton and colleagues, Neural Networks for Machine Learning lectures 11 and 12, https://www.cs.toronto.edu/~hinton/coursera/lecture11/lec11.pdf and https://www.cs.toronto.edu/~hinton/coursera/lecture12/lec12.pdf . The energy-probability relationship, conditional sampling, and learning phases were read. The two-unit calculation and fictional evaluation case are original; no historical benchmark claim is reproduced.

Previous: Boltzmann exploration Next: Bonferroni inequality