Does early exit reduce training time automatically?

No; its direct purpose is adaptive inference.

Does every input exit early?

No; some can reach the final head.

AI Glossary / Mathematics and optimization / Advanced

Early exit

Definition

Early exit is a deep-learning inference strategy that produces a prediction before an input has passed through every layer of a network. Intermediate prediction heads provide possible stopping points. An exit policy decides whether to accept a prediction at one of these points or continue computing. The purpose is to adapt computation per input while maintaining acceptable predictive performance.

What it means

Some inputs may be classified adequately using features computed near the beginning of a network. Others may need further processing. Early exit makes that distinction operational by attaching a prediction and a stopping decision to an intermediate representation.
It does not mean abandoning a prediction halfway through. The intermediate head must produce an output suitable for the task. Its apparent confidence also needs evaluation: a high score can accompany an incorrect answer. The resulting system includes the backbone, exit heads and routing policy, so evaluating only the final head misses how the deployed system actually behaves.

How it works

One approach adds classification branches at selected depths. Training supplies a learning signal to these branches, potentially alongside the final classifier. During inference, the system reaches a branch, computes its prediction and applies the exit criterion. If the criterion is not satisfied, processing continues.
BranchyNet provides an early example: it jointly trains exit branches and uses prediction entropy to decide whether an input can leave early. Lower entropy means the predicted distribution is more concentrated; it does not establish that the prediction is correct.
Other designs can use different criteria. Threshold selection should be separated from final performance evaluation. Cost accounting must include the intermediate heads and routing decisions, including the overhead paid by inputs that continue to the final exit. Fewer evaluated layers do not automatically translate into a proportional reduction in measured latency on a device.

Three practical examples

An image-classification service may accept confident intermediate predictions for some routine inputs and process others further. It evaluates errors among early exits as well as across the entire request population.
A device with limited computation can compare exit policies on representative inputs. The relevant question is the measured cost and quality of the complete policy, including difficult inputs and branch overhead.
A text classifier can use intermediate representations to make an early category prediction. A threshold selected on one type of text needs further evaluation when input topics, lengths or writing styles change.

Think of it this way

Think of a review process with several checkpoints. At each checkpoint, there is enough machinery to issue a decision, but the case can continue for further examination. The value comes from ending some reviews sooner without accepting too many mistaken decisions. A reviewer saying “I am confident” is evidence about the review process to test, rather than proof that every accepted decision is right.

Worked example

Consider an invented evaluation set of 100 classification requests. A baseline network uses ten computation units per request, for a total of 1,000 units. An early-exit version uses three units when a request exits at its intermediate head. A request that continues uses eleven units in total, including the intermediate head and routing overhead.
Under a chosen policy, sixty requests exit early and forty continue. Total computation is 60 × 3 + 40 × 11 = 620 units. Average computation is 6.2 units per request, compared with ten for the baseline. The reduction is (1,000 − 620)/1,000 = 38%.
Suppose checking against the evaluation labels finds two mistakes among early exits and three among continuing requests. The complete policy makes five mistakes, giving 95% accuracy on these 100 requests. If the baseline makes three mistakes on the same set, its accuracy is 97%. The policy saves computation in this illustration but loses two percentage points of accuracy.
Now consider a stricter policy that exits only forty requests early. Its computation is 40 × 3 + 60 × 11 = 780 units, or 7.8 per request. The reduction against the baseline is 22%. Suppose this policy makes one early-exit mistake and three final-exit mistakes, giving 96% overall accuracy.
These are hypothetical measured outcomes, not a theorem that stricter thresholds always improve accuracy. Both policies need their own evaluation. Neither computation count directly establishes a wall-clock speedup, because hardware execution and routing overhead affect elapsed time.

Illustrative case study

In a fictional service review, a team initially promotes the worked policy as “38% faster with confident answers.” The evaluation supports a 38% reduction in the defined computation units, but has not measured response time. It also shows five errors rather than the baseline's three.
The team revises the comparison to show computation, measured latency and predictive quality separately. It records the fraction of requests leaving at each head and checks whether the errors are concentrated in a particular input category. An overall accuracy figure could conceal poor performance for a less common category.
Before changing the threshold again, the team separates threshold-selection data from the final evaluation set. It then measures the complete deployed routing path. The goal is a supported operating choice, rather than treating a high exit rate as success by itself.

Common misconceptions and mistakes

Early exit is different from early stopping, which terminates training according to a stopping rule. Early exit changes how a trained network is evaluated for an input.
Do not ignore the cost of branches that are evaluated without being used for the final answer. Nor should an intermediate softmax score be presented as a guaranteed probability of correctness. Deeper processing is also not guaranteed to correct every early mistake; predictions can change in either direction.

Business use

Compare candidate exit policies using quality requirements, measured latency, computation and relevant operating conditions. Report the baseline and the full distribution of exit decisions. Reevaluate after material input or deployment changes. Savings are useful only when the resulting prediction service still meets its intended requirements.

Check your understanding

What is the first policy's average computation?

Answer: 6.2 units.

Why does a continuing request cost eleven rather than ten?

Answer: The example includes branch and routing overhead.

What was the accuracy tradeoff?

Answer: 95% for the first policy versus 97% for the baseline on the invented set.

Related terms

Inference
Entropy
Latency
Early stopping
intermediate classifier
adaptive computation
confidence calibration
deep neural network

Learn next

Compare exit policies using held-out labels and measured deployment costs, then examine calibration and distribution shift.

Sources

Aalto Dictionary of Machine Learning, early exit. https://aaltodictionaryofml.github.io/ADictML_Main.pdf Teerapittayanon, McDanel and Kung, BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks, 2016. https://www.eecs.harvard.edu/~htk/publication/2016-icpr-teerapittayanon-mcdanel-kung.pdf The numerical comparison and fictional service review are original teaching illustrations, not reported BranchyNet results.

Previous: Early-arriving fact Next: Early goal test