Does a low-loss path make the endpoint models equivalent?

No; models with similar loss can disagree on individual inputs.

Can a curve connect models with different architectures directly?

The usual weight-space construction assumes a compatible parameter space; additional mappings would require separate justification.

Is finding a path the same as sampling a Bayesian posterior?

No. A path-search objective does not automatically define or sample the posterior distribution.

AI Glossary / Probabilistic machine learning / Advanced

Mode connectivity

Definition

Mode connectivity describes the existence of a continuous path between trained model parameter settings along which a specified loss remains low. In neural-network research, it challenges the assumption that separately trained solutions must occupy isolated low-loss regions merely because a straight line between their weights crosses high loss.

What it means

Two training runs can produce different weight vectors with similar performance. Averaging those vectors directly may produce a poor model. That failure says something about the straight segment between the solutions, but it does not establish that every possible connecting route is poor.
Mode connectivity asks whether another route through parameter space keeps the loss low. The word “mode” commonly refers here to a trained solution or low-loss region; empirical endpoints need not be proven exact local minima. Connectivity is always relative to a model setup, a loss, and a tolerance for what counts as low.

How it works

Researchers can hold two trained endpoints fixed and optimize a connecting curve. Examples include a polygonal path with an adjustable bend or a quadratic Bezier curve with a learned control point. Candidate models along the path are evaluated using the chosen objective.
One practical procedure samples positions along a parameterized curve and updates its control parameters to reduce average training loss at those positions. This is a search procedure, not automatically a certificate that every point on the continuous path satisfies a loss bound.
Evaluation should inspect the path's loss profile and held-out behavior. Matching architecture and parameter conventions matter, as do evaluation details such as normalization statistics. A low training-loss path does not by itself establish that every intermediate model generalizes well or makes the same predictions as the endpoints.

Three practical examples

1. A research team could compare independently trained classifiers and investigate whether a poor weight average conceals an alternative low-loss connection.
2. An ensemble researcher could examine predictions from several points along a connection, checking whether they contribute useful diversity rather than assuming similar loss means identical outputs.
3. A model-development team could study how training choices affect the geometry between resulting solutions, while keeping the architecture, dataset, and measured objective explicit.
These are investigative uses. The existence of a path does not automatically provide a deployment-ready model or an effective ensemble.

Think of it this way

Imagine two camps on a valley floor with a hill directly between them. Walking in a straight line means climbing the hill, but a curved route might stay on the valley floor. The straight-line climb does not prove that the camps lie in separate valleys. Likewise, one low route does not prove that the entire surrounding landscape is flat.

Worked example

Consider the original illustrative loss L(x, y) = (x² + y² − 1)². This is a deliberately simple mathematical surface, not a measured neural-network loss. Points A = (−1, 0) and B = (1, 0) both have loss zero.
The straight path is s(t) = (−1 + 2t, 0), for t between zero and one. At t = 0.5, it passes through (0, 0), where the loss is one. At t = 0.25 and 0.75, the loss is 0.5625. The direct average of the endpoints therefore leaves the zero-loss region.
Now use the curved path c(t) = (−cos(pi × t), sin(pi × t)). It starts at A and ends at B. For every t, x² + y² = cos²(pi × t) + sin²(pi × t) = 1. Consequently, the loss is identically zero along the whole curve.
The maximum loss on the straight segment is one; on the curved path it is zero. Here the trigonometric identity proves the continuous-path claim, rather than merely checking a few sampled positions.
This example establishes the geometric distinction between a straight-line barrier and disconnected low-loss regions. It does not prove that arbitrary trained networks admit such a connection. It also demonstrates that averaging endpoint parameters is a different operation from following a learned path.

Illustrative case study

In a fictional scenario, Fern Classify trains two document classifiers with the same architecture but different random initializations. Both perform reasonably on the team's evaluation set, while their direct weight average performs poorly.
The researchers fit a connecting curve using training data and inspect losses along it. They then evaluate selected intermediate models on held-out data and examine their disagreements on individual documents. They keep test data out of the curve-fitting procedure.
The fictional lesson is to investigate the route and its intermediate models before drawing conclusions from an endpoint average. No successful connection, accuracy improvement, or ensemble benefit is asserted for this fictional team.

Common misconceptions and mistakes

Mode connectivity does not mean that every straight interpolation works. It does not imply convexity of the loss surface. A low average path loss can conceal a narrow high-loss section, so average and maximum are different summaries. Similar aggregate loss does not mean identical predictions on each example. An empirical result for particular networks is not a universal theorem for every architecture, dataset, or training objective. A two-dimensional visualization also shows only a restricted slice of a much larger parameter space.

Business use

The concept can help technical teams interpret variability between training runs and investigate ensemble or model-combination strategies. Decision-makers should ask what was actually demonstrated: a path with low training loss, acceptable held-out performance at tested positions, or a measurable benefit from combining predictions. These are distinct findings. Account for the computation required to find and evaluate paths, and compare any resulting approach with simpler baselines on the task that matters.

Check your understanding

Why does the failed average in the worked example not prove disconnection?

Answer: A curved zero-loss route still exists.

What proves that every point on that particular curve has zero loss?

Answer: The identity x² + y² = 1 along it.

Would checking ten points on an arbitrary learned curve provide the same proof?

Answer: No; unexamined sections could behave differently.

Related terms

Ensemble learning
Stochastic weight averaging
Loss landscape
local minimum
weight interpolation
Bezier curve
neural-network optimization

Learn next

Compare nonlinear mode connectivity with linear connectivity, then distinguish averaging weights from averaging model predictions.

Sources

Garipov and colleagues, Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018), Sections 3–4: https://papers.nips.cc/paper/2018/file/be3087e74e9100d4bc4c6268cdbe8456-Paper.pdf . The circular-loss example and Fern Classify scenario are original illustrations. Sections 3–4 independently consulted for sampled curve optimization and empirical loss profiles.

Previous: Mode Confusion Bias Next: Model