Does one pass give every partial derivative?

A single tangent seed gives one directional product. A full Jacobian generally needs multiple independent directions or an appropriate batched computation.

Does it perturb the input numerically?

No. It propagates derivative information through operation rules.

It applies exact differentiation rules to supported operations, subject to floating-point arithmetic, program approximations and conventions at nondifferentiable points.

AI Glossary / Probabilistic machine learning / Intermediate

Forward-mode automatic differentiation

Definition

Forward-mode automatic differentiation computes derivatives by carrying derivative information alongside values as a program evaluates a function. It applies the chain rule to each supported operation in the same direction as ordinary computation. A single seeded pass naturally computes a Jacobian-vector product: the output's local response to a chosen input direction.

What it means

A program can calculate a complicated quantity through many simple steps. Each step, such as multiplication or a sine function, has a known derivative rule. Forward mode combines those local rules automatically, avoiding the need to derive and maintain a separate expression for the whole derivative.

The ordinary value is often called the primal, and its accompanying directional derivative is called the tangent. A tangent is not a small numerical change that must be added to the input. It is derivative information propagated algebraically. This distinguishes automatic differentiation from finite differences, which estimate a derivative by comparing evaluations at nearby inputs.

How it works

Choose an input point x and a direction v. Seed each input with its corresponding tangent, then evaluate the program using paired value-and-tangent rules. For addition, add the tangents. For multiplication a times b, the output tangent is a's tangent times b plus a times b's tangent.

At the end, the primal output is f(x) and the tangent output is J_f(x)v, where J_f is the Jacobian matrix. There is no need to construct the whole Jacobian when only this product is required. Seeding a coordinate direction obtains the corresponding Jacobian column; multiple directions require additional work or a batched implementation.

Forward mode is often attractive when there are few input directions and many outputs. Reverse mode is often preferable for a scalar loss with many parameters. Actual cost also depends on program structure, available derivative rules and implementation. Automatic differentiation follows the implemented computation, including any approximations in that computation.

Three practical examples

1. Engineering sensitivity: assess how several simulated outputs respond locally to one design parameter. A directional derivative can reveal which outputs change most without constructing a large full derivative matrix.

2. Model diagnostics: measure the local response of multiple predictions to a specified input perturbation direction. Treat the result as mathematical sensitivity, not proof of a causal effect in the real world.

3. Numerical optimization: compute Jacobian-vector or higher-order directional products inside an algorithm that needs these products rather than every derivative entry. This can avoid unnecessary matrix storage.

Think of it this way

Think of a calculation traveling with a companion note explaining how it would respond to one infinitesimal input movement. Each operation updates both the value and the note. The analogy captures paired propagation. The note is a derivative, not a record of actual measurement error or a guarantee that a large input change will behave the same way.

Worked example

Take f(x, y) = x times y + x squared at the point x = 2, y = 3. Choose direction v = (1, −2), meaning that the derivative is taken along x increasing at rate one and y decreasing at rate two.

Seed x with tangent 1 and y with tangent −2. For a = x times y, the primal is 6 and its tangent is 1 × 3 + 2 × (−2) = −1. For b = x squared, the primal is 4 and its tangent is 2 × 2 × 1 = 4. Adding them gives f = 10 and output tangent 3.

Check independently: the gradient is (y + 2x, x) = (7, 2), and its dot product with (1, −2) is 7 − 4 = 3. Along a finite move h, f(2 + h, 3 − 2h) equals 10 + 3h − h squared. The tangent supplies the first-order term 3h, not the entire finite-change formula. No finite-difference step size was needed to propagate the derivative.

Illustrative case study

Fictional case: Pine Design maintains a simulation with one adjustable material parameter and many reported outputs. An engineer adds forward-mode differentiation to quantify local output sensitivities and checks a small simplified case against hand calculations.

The team then inspects operations where the simulator clips values or changes branches. It reports the point and parameter range associated with each sensitivity rather than presenting one derivative as a global relationship. This fictional example illustrates an implementation review, not a claim that automatic differentiation validates the physical simulation.

Common misconceptions and mistakes

Forward mode is not merely the forward evaluation pass used before neural-network backpropagation. It is a derivative-propagation method. It also differs from symbolic differentiation, which manipulates expressions to produce derivative expressions.

Automatic differentiation avoids finite-difference truncation error but still operates with finite-precision arithmetic and implemented derivative conventions. Nondifferentiable operations may use a chosen subgradient or have no supported rule. Branch boundaries, discrete choices and external code require care. A numerically returned derivative can be faithfully computed for a program whose model assumptions are wrong.

Business use

When reviewing a sensitivity feature, ask what input direction was seeded, what output was differentiated and at which operating point. These details determine the meaning of the result. A derivative with respect to a normalized feature also needs conversion before it can be interpreted in original units.

Benchmark the products actually needed rather than assuming one mode is universally faster. Validate representative simple cases and inspect custom derivative rules. If business decisions involve large changes, supplement local derivatives with scenario evaluations because curvature and changes in operating regime may matter.

Check your understanding

What does the output tangent 3 mean in the worked example?

Answer: At (2, 3), moving in direction (1, −2) changes the output locally at rate 3. It is not the derivative with respect to x alone and does not describe all finite movements exactly.

Related terms

Automatic differentiation
Jacobian-vector product
Chain rule
Gradient
reverse-mode differentiation
tangent vector
dual number
finite difference

Learn next

Propagate values and tangents through a short expression using two different seed directions. Then compare forward and reverse mode on a function with many outputs and a scalar loss, focusing on which derivative products each task actually needs.

Sources

JAX authors, Forward- and reverse-mode autodiff in JAX: https://docs.jax.dev/en/latest/jacobian-vector-products.html

JAX authors, jax.jvp reference: https://docs.jax.dev/en/latest/_autosummary/jax.jvp.html

Independent review consulted official JAX JVP rules, directional-product and interface documentation. Original arithmetic verified; no runtime benchmark is claimed.

Previous: Forward mapping Next: Forward operator