Does it require deep learning?

No. Ordinary numerical formulas can form a differentiable program.

Is automatic differentiation the optimizer?

No. It supplies derivative information that an optimizer may use.

Does a gradient measure a real-world causal effect?

Not automatically. It measures local change within the specified mathematical computation.

AI Glossary / Graphs and symbolic AI / Advanced

Differentiable programming

Definition

Differentiable programming is an approach to designing numerical programs whose outputs can be differentiated with respect to selected inputs or parameters. It enables gradient-based adjustment of program parameters, often using automatic differentiation. The program can combine learned components with explicitly written calculations; it need not consist only of a conventional neural network.

What it means

A program normally computes an output from inputs and settings. A differentiable program also supports calculating how that output, or an associated objective, changes locally as selected quantities change. These derivatives can guide parameter updates or help examine sensitivity.
The important design choice is the computation being differentiated. Developers still choose the model, data, objective, and adjustable parameters. Automatic differentiation does not discover whether those choices represent the intended problem. A precisely calculated gradient of an unsuitable objective can guide the program toward an unsuitable result.

How it works

A numerical computation is composed of operations with derivative rules. Automatic differentiation combines those rules through the chain rule to calculate derivatives of the overall computation. It works with the program's operations rather than estimating every derivative by repeatedly perturbing inputs.
A scalar loss can compare the program's output with a target. An optimizer uses the loss gradient to update selected parameters, then the program runs again with the revised values. Computing a gradient and choosing an optimization step are separate operations; the derivative does not specify an automatically safe step size.
Programs also contain complications such as discrete choices, external routines, and nonsmooth operations. A hard threshold can be locally constant away from its jump and have no ordinary derivative at the jump. Replacing it with a smooth approximation changes the modeled computation. Such changes require justification and evaluation, not merely successful execution of an autodiff tool.

Three practical examples

Sensor calibration: A team writes a numerical model converting raw readings into an estimated quantity. It differentiates an error measure with respect to calibration parameters and adjusts them using reference measurements. Good fit on calibration data still needs independent validation.
Simulation fitting: A researcher combines a physical calculation with a learned correction component. Gradients can connect the final mismatch to parameters in both parts, provided the relevant operations support the required differentiation.
Image processing: A developer adjusts parameters of a differentiable image transformation against an explicit reconstruction loss. Lower loss measures improvement under that criterion, which may not capture every feature that viewers consider important. Evaluation should include the intended visual task.

Think of it this way

Imagine a calculation built from connected adjustment knobs. Differentiation traces how a small turn of each knob would affect the final score through all the intermediate steps. Optimization decides which turns to make. Neither operation establishes that the score measures the outcome people actually want.

Worked example

Consider a small calibration program with input x = 2, adjustable gain a, intermediate value z = ax, and output y = z squared. The target output is 9. Define the loss as L = (y - 9) squared / 2. Start at a = 1.
The forward calculation gives z = 2, y = 4, and L = 25/2 = 12.5. To differentiate through the program, compute dL/dy = y - 9 = -5, dy/dz = 2z = 4, and dz/da = x = 2. Multiplying these local derivatives gives dL/da = -5 × 4 × 2 = -40.
With a learning rate of 0.01, a gradient-descent update gives a_new = 1 - 0.01 × (-40) = 1.4. Running the program again yields z = 2.8, y = 7.84, and L = (7.84 - 9) squared / 2 = 0.6728. The selected update reduces this example's loss.
However, a learning rate of 0.1 would give a_new = 5, y = 100, and L = 4140.5. The same correct derivative can therefore lead to a much worse result when the step is too large.
There is also an identification issue: both a = 1.5 and a = -1.5 produce output 9. A perfect fit to this single observation does not determine which gain is appropriate. Additional measurements or justified parameter constraints may be needed.

Illustrative case study

Fictional Maple Instruments builds a calibration workflow from a known numerical transformation and a small learned correction. Its first review focuses only on whether the software returns gradients and lowers the training loss.
The team expands the review to compare derivative calculations with small numerical perturbations on representative smooth cases. It checks parameter ranges and evaluates the calibrated program on measurements excluded from fitting. It also investigates whether multiple parameter settings explain the same observations.
The resulting workflow treats gradient availability as one technical capability within a larger modeling process. This fictional scenario illustrates validation choices and does not claim a measured improvement for a real instrument.

Common misconceptions and mistakes

“Differentiable programming means every program can be trained unchanged.” Unsupported or discrete operations can require redesign.
“Automatic differentiation is finite-difference estimation.” The methods calculate derivatives differently, although finite differences can help check smooth examples.
“A correct gradient guarantees a better next result.” The step size and objective landscape matter.
“Differentiable means interpretable or correct.” It describes derivative behavior, not overall model quality.
“Training loss identifies the true parameters.” Different settings can produce the same predictions.

Business use

Consider this approach when a task has meaningful numerical parameters and an objective that can support useful derivatives. Specify what may change, what must remain constrained, and how success will be measured beyond the training calculation.
Check gradient behavior and computational cost before committing to a large implementation. Retain independent outcome tests, especially when smooth approximations replace operational rules or when the data cannot uniquely determine the parameters.

Check your understanding

What is the starting loss gradient with respect to a?

Answer: It is -40.

Why does the larger update fail in the example?

Answer: It moves far beyond the locally useful change and greatly increases the loss.

Why can zero loss leave uncertainty about the gain?

Answer: Positive and negative gain values can produce the same squared output.

Related terms

Learn next

Write a short numerical program with two adjustable parameters. Differentiate its loss, compare against finite differences at smooth points, and test how step size and parameter constraints affect the resulting updates.

Step size

Sources

AIMA glossary, differentiable programming: https://github.com/aimacode/aima-glossary/blob/master/glossary.md JAX documentation, automatic differentiation and gradient checks: https://docs.jax.dev/en/latest/automatic-differentiation.html JAX FAQ, discontinuities and zero gradients: https://docs.jax.dev/en/latest/faq.html?highlight=nan Blondel and Roulet, The Elements of Differentiable Programming, overview: https://arxiv.org/abs/2403.14606

Previous: Differentiable neural computer Next: Differential entropy