Does gradient checkpointing save a restart file?

Not by itself. Persistent recovery checkpoints are a separate mechanism.

Does it change the model architecture?

It need not; it changes how intermediate computation is stored and recreated.

Are gradients guaranteed identical in every implementation?

No. Equivalent computation requires correct handling of the function, state, randomness, and applicable implementation limitations.

AI Glossary / MLOps and deployment / Intermediate

Gradient Checkpointing

Definition

Gradient checkpointing, also commonly called activation checkpointing, is a training technique that reduces stored intermediate activations by recomputing some of them when gradients are needed. It trades additional computation for lower activation-memory use. Unlike a checkpoint saved for recovery, it is not primarily a way to resume training after a machine failure.

What it means

During a neural network's forward pass, intermediate values help calculate later gradients. Keeping many of these values available can consume substantial memory, especially with deep networks, long inputs, or large batches.
Checkpointing changes what is retained. Selected inputs or boundaries remain available, while some internal values are recreated during the backward calculation. The intended mathematical training computation can remain the same when recomputation reproduces the necessary behavior correctly.
The name can be misleading: the main target is usually stored activations, not a periodic archive of gradient files. It also does not remove memory needed for model parameters, optimizer state, or other allocations.

How it works

Choose regions of the model whose intermediate values are expensive to retain relative to the cost of recomputing them. During the original forward pass, preserve enough information to reconstruct the needed values later.
When backward propagation reaches a checkpointed region, rerun the relevant forward computation and use the recreated values to calculate derivatives. Implementations differ in how much is recomputed and when they can stop.
Correctness requires attention to changing state and randomness. If a recomputed function follows a different path or uses inconsistent random behavior, its gradients may not match the intended original computation. Framework support can manage some state, but its documented limitations still matter.

Three practical examples

A team training a deep network can checkpoint selected blocks when stored activations are a major contributor to peak memory.
A long-sequence experiment can assess whether recomputation permits its intended input length within the available memory budget.
A researcher can compare alternative checkpoint regions, measuring both memory and step time before selecting a configuration for a larger run.

Think of it this way

Think of solving a long calculation on a small whiteboard. Instead of keeping every intermediate line visible, you retain a few useful points and redo parts when needed. This saves board space but costs work. The analogy captures the trade-off, although automatic differentiation requires particular intermediate quantities and careful consistency that casual handwritten reconstruction does not.

Worked example

Consider a simple invented computation with scalar input x. A first region calculates a = 2x and b = a². A later operation calculates loss L = 3b. For x = 2, the forward values are a = 4, b = 16, and L = 48.
The derivative follows the chain: dL/db = 3, db/da = 2a = 8, and da/dx = 2. Therefore dL/dx = 3 × 8 × 2 = 48.
Suppose the intermediate a is not retained after the first forward pass, but the input x remains available. During backward calculation, recomputing a = 2x recovers a = 4, allowing the same derivative calculation. This scalar example demonstrates the mechanism; saving one scalar would not justify meaningful checkpointing overhead in practice.
For a separate illustrative resource comparison, suppose a measured training configuration peaks at 18 GB, of which the team's analysis attributes 10 GB to activation-related storage and 8 GB to other required memory. A hypothetical checkpoint configuration reduces that activation component to 6 GB while leaving the other component unchanged. The simplified total becomes 14 GB, a reduction of 4 GB or about 22.2% of the original 18 GB.
Those resource numbers are invented, not a benchmark. Actual peak allocations can overlap differently, so component arithmetic is a planning illustration rather than a substitute for measuring a running implementation.

Illustrative case study

In a fictional language-model experiment, a team enables activation checkpointing and finds that a larger input now fits into memory. However, the training step takes longer because some forward computation repeats.
The team compares total experiment time and checks gradients on a small controlled case before committing to the new configuration. It keeps separate persistent recovery checkpoints, recognizing that the memory technique does not preserve the job after a process failure.
The lesson is that fitting into memory, computing the intended gradients, and recovering interrupted work are distinct requirements.

Common misconceptions and mistakes

Do not confuse activation checkpointing with saving model and optimizer state to durable storage. The two techniques address different problems and can be used together.
Do not expect it to reduce every source of memory usage. If parameters or optimizer state dominate, activation savings may not solve the bottleneck.
Do not assume more checkpointing is always better. Recomputing expensive regions can reduce throughput substantially.
Do not overlook side effects, mutable global state, or randomness inside recomputed functions. A function that behaves differently the second time can undermine gradient correctness. Use the framework's supported pattern and validate the actual configuration rather than assuming equivalence from the technique's name.

Business use

Gradient checkpointing offers an engineering option when memory limits prevent a useful training configuration. Evaluate it through a resource budget: peak memory, step time, feasible input size, and total time to the intended training outcome.
Separate a memory feasibility benefit from a cost-efficiency claim. A slower step might still enable an otherwise impossible experiment, but that does not mean every checkpoint configuration reduces total cost. Record the chosen regions and framework settings alongside other experiment parameters so performance comparisons remain interpretable.

Check your understanding

A run uses activation checkpointing but never saves training state to durable storage. Can it necessarily resume after its process disappears?

Answer: No. Recomputation for memory savings does not provide a persistent recovery snapshot.

Related terms

Learn next

Profile a small training run to identify its main memory components. Then compare one supported checkpoint configuration with the baseline, checking gradients, peak memory, and elapsed step time before scaling up.

Checkpoint

Sources

Sources: NVIDIA AI Enterprise glossary, Gradient Checkpointing: https://docs.nvidia.com/ai-enterprise/release-8/latest/reference/glossary.html PyTorch checkpoint documentation, recomputation and state limitations: https://docs.pytorch.org/docs/2.14/checkpoint.html

Previous: Gradient boosting Next: Gradient clipping