Does it remove the smallest weights?

Not necessarily. Its ranking combines parameter size with estimated local curvature.

Why train again after pruning?

The retained parameters may compensate for removals, and the model has moved away from its previous trained configuration.

Is the estimated deletion cost exact?

Generally no. It depends on approximations that omit higher-order changes and interactions between parameters.

AI Glossary / Graphs and symbolic AI / Intermediate

Optimal brain damage

Definition

Optimal Brain Damage is a neural-network pruning method that estimates how much a trained network's objective would increase if particular parameters were removed. Introduced by Yann LeCun, John Denker, and Sara Solla, it uses approximate second-derivative information to rank parameters by their estimated importance. Parameters with low estimated impact can be removed, followed by further training.

What it means

A network may contain more adjustable parameters than it needs for a particular task. Removing some can make its representation smaller. The challenge is deciding which removals are least damaging to the behavior that training has established.

Optimal Brain Damage addresses that decision through local sensitivity. A small parameter is not necessarily unimportant: the objective may be very sensitive to changes in it. A larger parameter can sometimes be less consequential if the objective changes slowly along that parameter direction. Despite the historical name, the method does not guarantee a globally optimal compressed network.

How it works

Start with a trained network and a differentiable objective. A Taylor approximation describes how the objective changes when parameters change. Near a local minimum, the first-derivative contribution is assumed to be approximately zero.

The method further approximates the objective's curvature using diagonal second derivatives, leaving out interactions between different parameters. Setting a parameter w to zero changes it by minus w. Under these assumptions, its estimated deletion cost, or saliency, is one half of its diagonal curvature multiplied by w squared.

Compute these estimates, identify low-saliency parameters, remove selected parameters, and retrain. Repeat as appropriate. Retraining matters because the remaining parameters can adjust after removal. The approximation can become unreliable when pruning moves the model far from the point where its curvature was estimated or when ignored parameter interactions are substantial.

Three practical examples

Model compression: a team compares curvature-based parameter rankings with magnitude-only rankings before deciding how to reduce a trained network.
Research education: students use a small differentiable model to see why two equally sized weights can have different deletion costs.
Deployment preparation: engineers prune a candidate network, then separately measure predictive quality, memory use, and actual runtime on the intended hardware.

Think of it this way

Think of removing supports from a temporary display after estimating how much each removal would disturb its shape. A small support could matter greatly, while a larger one might be redundant. The analogy illustrates sensitivity, not structural engineering: neural-network parameters interact mathematically, and a diagonal curvature estimate can overlook their combined effects. It is not a safety procedure for physical structures.

Worked example

Consider three parameters in a hypothetical trained model. Their values are 0.1, 0.5, and 0.2. Suppose the corresponding diagonal curvature estimates are 100, 1, and 4.

Their estimated saliencies are:
First: 0.5 × 100 × 0.1 squared = 0.5.
Second: 0.5 × 1 × 0.5 squared = 0.125.
Third: 0.5 × 4 × 0.2 squared = 0.08.

The third parameter has the smallest estimated deletion cost. A magnitude-only rule would instead select the first parameter, whose absolute value is smallest. This example demonstrates why curvature can change the ranking.

These numbers are estimated objective increases, not percentages of accuracy lost. They assume the stated local approximation. Removing the second and third parameters together would give an additive estimate of 0.205 under the diagonal approximation, but their actual joint effect could differ because the calculation excludes cross-parameter interactions. Validation and retraining must establish what the modified model actually does.

Illustrative case study

Fictional case: Cedar Instruments develops a small classifier for labeling recordings. Its researchers want to reduce model storage while retaining performance on a held-out evaluation set.

They estimate parameter saliencies, remove a conservative group of low-ranked parameters, and retrain the remaining network. A separate evaluation checks errors across recording conditions rather than relying only on the training objective.

The engineering team also measures the deployed representation. It finds that zero-valued parameters will require a suitable storage and execution format before they can provide useful deployment savings. The case illustrates an evaluation workflow; no measured accuracy gain, compression ratio, or speed improvement is claimed.

Common misconceptions and mistakes

Do not interpret the word optimal as a proof that no better pruning decision exists. The method relies on local and diagonal approximations.

Do not equate saliency with weight magnitude, and do not read an estimated objective change as a direct accuracy change. These are different quantities.

Avoid assuming that many zeros automatically make inference faster. Runtime depends on representation, hardware, software, and the pattern of sparsity. Finally, the objective used to calculate saliency may not capture every important behavior. Rare cases and particular user groups still need independent evaluation.

Business use

For a business team, the useful question is whether pruning yields a model that meets a defined deployment objective at an acceptable quality level. That could involve storage, energy, latency, or maintainability, but each benefit needs measurement.

Keep the baseline model, evaluation data, pruning decisions, and retraining settings traceable. A smaller parameter count alone does not establish a better product. Compare the complete deployment, including any additional processing or sparse-format overhead.

Check your understanding

A small parameter has high curvature while a larger parameter has low curvature. Must the smaller parameter be removed first?

Answer: No. Compare their saliencies using both curvature and squared parameter value. Magnitude alone does not determine the method's ranking.

Related terms

Pruning
Hessian
Saliency
Sparsity
Retraining
Objective function
second-order approximation
model compression

Learn next

calculate saliencies for a small hypothetical parameter table, then vary the curvatures while keeping the weights fixed. Study Hessian cross terms to understand what the diagonal approximation omits. Finally, compare theoretical sparsity with measured storage and inference behavior.

Sources

Primary source: LeCun, Denker, and Solla, “Optimal Brain Damage,” NeurIPS 1989: https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf Relevant method and approximation passages were consulted. Numerical examples and Cedar Instruments are original illustrations. The supplied category is retained; this entry concerns neural-network pruning.

Previous: Optical neural network Next: Optimal controllers