Should every training step create a checkpoint?

Not necessarily. Frequency should balance recovery needs against overhead and storage cost.

Does checkpointing eliminate all lost work?

No. Progress since the latest recoverable snapshot may need to be repeated.

Can saving run in the background?

Some systems support asynchronous or overlapping saves, but they must still capture consistent state and report when a snapshot is actually recoverable.

AI Glossary / MLOps and deployment / Beginner

Checkpointing

Definition

Checkpointing is the process of saving a computation's recoverable state at selected points so that work can resume after interruption. In machine-learning training, it commonly creates persistent snapshots of model and training state. Its purpose is to limit lost progress and support recovery. This fault-tolerance meaning is different from gradient or activation checkpointing, which saves memory by recomputing intermediate activations.

What it means

A checkpoint is the saved artifact; checkpointing is the policy and mechanism that creates, stores, validates, and restores such artifacts. A training job may save every fixed number of steps, after a time interval, or at meaningful boundaries.
More frequent saves can reduce how much work must be repeated after failure, but they consume storage bandwidth, time, and capacity. Less frequent saves reduce that overhead while exposing more recent progress to loss.
Recovery depends on the entire path. A snapshot stored only on a machine that disappears may not survive the failure it was meant to address. A durable file without compatible state or a functioning restore procedure is also insufficient.

How it works

Identify the failure situations to recover from and the state required for a valid continuation. Capture that state consistently, write it to the intended durable location, and mark the new checkpoint available only after its required pieces are complete.
Retain suitable earlier checkpoints so a failed or corrupt newest save does not erase the only recovery option. In distributed jobs, the pieces must correspond to a coherent state rather than an arbitrary mixture of worker progress.
On restart, select a valid checkpoint, restore the required components, and reconcile progress with data processing and external effects. Repeating computation can also repeat outputs unless the surrounding workflow is designed to recognize or safely handle replay.

Three practical examples

A long model-training run can periodically save state to recover from a worker or infrastructure interruption.
A distributed computation can coordinate snapshots so that restoring several workers does not combine incompatible moments of execution.
An experimental workflow can test recovery deliberately on a small run, verifying that its checkpointing policy protects the intended progress rather than merely creating files.

Think of it this way

Think of saving a long document as you work. Frequent saves protect recent edits, but saving a large document can interrupt work, and saving only to a failing drive may not help. The analogy explains the basic trade-off, though a training process has interacting state such as optimizer buffers and data position that a simple document does not.

Worked example

Consider an invented job that performs 40 minutes of useful computation, then pauses for a two-minute checkpoint save. One complete cycle takes 42 minutes. Ignoring other overhead, the fraction of wall time spent saving is 2/42, approximately 4.76%.
If an interruption occurs uniformly during the 40-minute compute portion and the previous checkpoint is valid, the mean lost compute is 20 minutes. An interruption 31 minutes into that portion loses 31 minutes, plus whatever time recovery itself requires.
Now save after every 10 minutes of useful computation, still taking two minutes per save. The save fraction becomes 2/12, approximately 16.67%, while the same conditional uniform-failure assumption gives mean lost compute of five minutes.
The second policy reduces expected lost work under these assumptions but spends a larger share of normal runtime saving. These figures do not identify a universally optimal interval. They omit failures during saves, changing failure rates, overlapping writes, restore time, and storage contention.
The example's purpose is to make the trade-off measurable. A real policy should use observed job behavior and tested recovery rather than copying the illustrative interval.

Illustrative case study

In a fictional research project, a training run writes a new checkpoint over its only previous snapshot. An interruption during that write leaves no usable restart point.
The team revises the process to preserve the last validated snapshot while preparing the new one. It tests the failure path with a small job and checks that the latest incomplete save is not selected for recovery.
The lesson is that save frequency alone does not establish resilience. Publication, retention, and restore selection must work together. The scenario does not prescribe one storage API or claim that every storage system provides the same atomicity guarantees.

Common misconceptions and mistakes

Do not assume a checkpoint saved locally survives loss of the host or its storage. Match the location and durability to the failure model.
Do not overwrite the only recoverable state before the replacement is complete. Likewise, do not combine model weights from one step with optimizer or scheduler state from another without a justified recovery design.
Do not equate restart with exact replay. Randomness, data ordering, software differences, and nondeterministic computation can change the continuation.
Finally, do not confuse fault-tolerance checkpointing with activation checkpointing. The latter trades extra computation for reduced training memory and is not a substitute for persistent recovery snapshots.

Business use

Checkpointing supports predictable recovery costs for expensive or interruption-prone jobs. Define an acceptable amount of lost progress, measure save and restore overhead, and test the recovery procedure in the environment that will use it.
Also account for retained storage, access controls, artifact identification, and cleanup rules. For workflows with external writes, confirm how replayed steps avoid unintended duplicate effects. The useful outcome is a tested ability to resume meaningful work, not simply a large count of saved files.

Check your understanding

A checkpoint captures step 1,200 and is fully saved. The job fails after step 1,260. How many completed steps are absent from that snapshot?

Answer: Sixty steps. Restarting from it may require repeating those steps, with additional restore overhead and any replay handling required by the workflow.

Related terms

Checkpoint
Fault tolerance
persistent storage
recovery
optimizer state
distributed training
restart
idempotency
activation checkpointing

Learn next

Measure checkpoint size, save time, and restore time on a small representative run. Simulate interruption before, during, and after a save, and verify which snapshot is selected. Then choose an interval and retention policy based on the recovery behavior actually observed.

Checkpoint

Sources

Aalto Dictionary of Machine Learning, checkpointing entry: periodic persistent state capture for fault tolerance. https://aaltodictionaryofml.github.io/ADictML_Main.pdf PyTorch, Saving and Loading Models: state components needed for general training checkpoints. https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html

Previous: Checkpoint Next: Chen distribution