Is the chain rule the same as gradient descent?

No. The chain rule computes derivatives through compositions; gradient descent uses gradients to choose parameter updates.

Why do some derivative contributions add?

When a variable influences the output through several intermediate dependencies, the total derivative includes every path's contribution.

Does automatic differentiation remove the need to understand the graph?

No. The system follows the calculation provided, so missing or unintended dependencies still matter.

AI Glossary / Mathematics and optimization / Intermediate

Chain rule of calculus

Definition

The chain rule of calculus gives the derivative of a composite function: a function whose input is the output of another function. For y=f(g(x)), it states that dy/dx=f′(g(x))g′(x), when the required derivatives exist. It is a foundation of gradient calculations in machine learning.

What it means

A model often transforms an input several times before producing an output or a loss. The chain rule connects the local rate of change at each stage to the rate of change of the overall calculation.

The crucial detail is where each derivative is evaluated. The derivative of the outer function is evaluated at the current output of the inner function. Simply multiplying two derivative formulas without respecting their inputs can give an incorrect answer. For calculations with branching dependencies, contributions from every relevant path must be included.

How it works

For a simple chain x→u→y, a small change in x changes u by approximately (du/dx) times that change. The resulting change in y is approximately (dy/du) times the change in u. Multiplying these local factors gives the overall derivative.

If y depends on several intermediate quantities u_1 through u_m, all of which depend on x, the derivative adds their contributions: dy/dx is the sum of (partial y/partial u_i)(du_i/dx). Vector-valued compositions use compatible Jacobian matrices, with multiplication order determined by the chosen convention.

Backpropagation applies this principle efficiently through a computational graph. It reuses intermediate results and accumulates contributions where paths meet. The chain rule supplies the mathematical relationship; the differentiation procedure organizes its computation, and an optimizer subsequently decides how to change parameters.

Three practical examples

1. Neural-network training: calculate how a loss changes with an early layer's weight through the later transformations that depend on it.

2. Sensitivity analysis: assess how a prediction changes when an input first passes through a scaling or differentiable feature transformation.

3. Model debugging: compare a hand-derived gradient of a small composed calculation with the derivative produced by an automatic differentiation system.

Think of it this way

Think of connected adjustment dials: the first moves an intermediate setting, which changes a final indicator. The overall local response combines how strongly each stage reacts.

This analogy is about small changes near the current settings. Large changes may encounter different local responses, and a branching mechanism requires adding the contributions of its separate paths. A derivative is not a universal conversion factor valid everywhere.

Worked example

Consider a scalar prediction z=wx+b and loss L=(z−t)²/2. Let x=3, w=2, b=−1, and target t=4. The prediction is z=2×3−1=5, so the loss is (5−4)²/2=0.5.

The outer derivative is dL/dz=z−t=1. The inner derivative with respect to w is dz/dw=x=3. Therefore dL/dw=1×3=3. Likewise, dL/db=1×1=1. Holding x, b, and t fixed, a small weight increase of 0.001 predicts a loss increase of approximately 3×0.001=0.003.

Direct substitution gives a new prediction of 5.003 and loss 1.003²/2=0.5030045. The actual increase, 0.0030045, differs slightly from the linear approximation because of the quadratic term.

For a branching check, let y=x²+x×x. Each branch contributes 2x, so the total derivative is 4x. Ignoring one path incorrectly gives 2x. These examples are hand calculations, not a reported software execution.

Illustrative case study

Fictional case: a team builds a custom loss for an inventory forecast. Its hand-written gradient initially follows the prediction branch but omits a second branch containing a penalty on the same parameter.

The team draws the computation graph, traces both dependencies, and adds their derivative contributions. It checks the corrected result on a small numerical example before using the operation in training. This addresses a gradient implementation error; it does not establish that the chosen penalty reflects the right business objective or that training will find a global minimum.

Common misconceptions and mistakes

A common mistake is forgetting the inner derivative, as when differentiating (3x+1)² as only 2(3x+1). The missing factor is 3. Another is overlooking multiple paths from the same variable to the output.

Ordinary chain-rule formulas assume the relevant differentiability conditions. A software convention at a nondifferentiable point is not proof that an ordinary derivative exists there. Automatic differentiation also differentiates the recorded calculation, including its chosen operations; it cannot determine whether that calculation represents the intended model.

Business use

The chain rule helps explain what training gradients measure and why connected computations must be specified correctly. For teams developing custom model components, small derivative checks can reveal errors that otherwise appear as unexplained training behavior.

For decision makers interpreting sensitivity, keep the local scope clear. A derivative shows the response near a particular input and parameter setting. It does not alone establish causation, quantify every uncertainty, or predict the effect of a large intervention outside that neighborhood.

Check your understanding

If y=(2x−3)³, what is dy/dx at x=2?

Answer: The derivative is 3(2x−3)²×2=6(2x−3)². At x=2, it equals 6. The final factor of 2 comes from differentiating the inner expression.

Related terms

Learn next

draw computational graphs with shared inputs, then practice adding contributions at branching paths. Connect the scalar chain rule to Jacobian multiplication and compare symbolic, automatic, and finite-difference differentiation, including their different limitations.

Sources

Dive into Deep Learning, Calculus, section on the chain rule: https://d2l.ai/chapter_preliminaries/calculus.html . The scalar and multivariable formulas and the discussion connecting them to backpropagation were read. The loss calculation, finite-change comparison, branching example, and fictional debugging scenario are original.

Previous: Chain rule Next: Chaining (vector processing)