AI Glossary / Generative AI and language / Advanced

Zero Redundancy Optimizer

Definition

Zero Redundancy Optimizer, usually written ZeRO, is a family of distributed-training memory optimizations that partitions model states across data-parallel processes instead of keeping every state fully replicated on every device. Its stages progressively partition optimizer states, gradients and model parameters. The aim is to reduce redundant memory use so larger training workloads can fit within the available hardware.

What it means

Training stores more than a model's visible weights. It can also retain gradients and optimizer information, such as Adam's running estimates. In conventional data-parallel training, each participating process commonly holds copies of these states while working on different training examples.
ZeRO distributes responsibility for those states. Each process keeps a portion, and communication makes the necessary information available for computation and updates.
The name does not mean that training uses zero memory or that all forms of duplication disappear. Activations, temporary buffers and other working storage remain important.

How it works

ZeRO Stage 1 partitions optimizer states. Stage 2 also partitions gradients. Stage 3 additionally partitions model parameters, gathering the required parameters for computation and managing their partitioned storage.
Moving to a higher stage changes which state categories are distributed; it is not simply a request to reduce every memory allocation by the same percentage.
Offloading is a related but separate choice. It moves selected states to memory such as CPU RAM or storage rather than keeping everything on the accelerator. Partitioning answers which process owns a state; offloading answers where that state is stored.
The benefit depends on the workload and communication system. Memory savings can enable training that otherwise fails to fit, but additional communication and data movement must be considered when measuring throughput.

Three practical examples

• A research team cannot fit a model's optimizer states on each accelerator. It evaluates Stage 1 before changing the architecture, measuring both memory and training-step time.
• A larger training job uses Stage 3 to distribute parameters as well as gradients and optimizer states. The team checks temporary parameter gathering when estimating peak memory.
• An experiment combines state partitioning with offloading. Its evaluation includes transfer time and host-memory requirements rather than reporting accelerator memory alone.

Think of it this way

Imagine four colleagues who each keep an identical large reference collection. They could divide the collection into four parts and exchange the material needed for each task.
Each desk holds less permanently, but sharing has a cost, and a colleague may temporarily need several borrowed volumes. This analogy explains both the savings from avoiding full copies and why the largest momentary workspace requirement can exceed the size of one stored share.

Worked example

Consider an original simplified memory calculation for one billion parameters across four processes. Assume two bytes per parameter for model weights, two for gradients, and twelve for optimizer-related storage: a four-byte master weight plus two four-byte Adam moments.
Without partitioning, these model states occupy 2 + 2 + 12 = 16 billion bytes per process, or 16 decimal GB. The calculation concerns these states only.
At Stage 1, optimizer-related storage is divided by four, while weights and gradients remain replicated: 2 + 2 + 12/4 = 7 GB per process.
At Stage 2, gradients are also divided: 2 + 2/4 + 12/4 = 5.5 GB. At Stage 3, all three counted categories are partitioned: (2 + 2 + 12)/4 = 4 GB.
Across four processes, the corresponding persistent-state totals are 64, 28, 22 and 16 GB. This explains the reduction in replicated storage under the stated assumptions.
It does not predict total peak accelerator memory. Activations, communication buffers, temporary gathers and implementation details add requirements. Nor are these numbers universal across optimizers or precision configurations. Dividing the idealized state total by four is not proof that a four-GB device can train the model.

Illustrative case study

Fictional Willow Research estimates that its selected ZeRO configuration should substantially reduce model-state storage. The team initially expects the training job to fit comfortably, but the run still reaches the device-memory limit.
Profiling shows that long input sequences and temporary working buffers account for much of the remaining peak. The team distinguishes persistent states from activations and transient allocations before comparing revised configurations.
It also records training-step time. One configuration fits with more headroom but runs slower because data movement becomes more expensive. The eventual choice reflects the required model, sequence length and acceptable training cost, rather than assuming the highest ZeRO stage must always be the best operational setting.

Common misconceptions and mistakes

ZeRO is not a compression method that removes model parameters. Partitioning changes where states reside while preserving the distributed training computation's intended role.
Stage 1 does not partition every model-state category. Its defining change concerns optimizer states; later stages extend the partitioning.
A memory reduction is not automatically an equal speed increase. Communication, computation and memory capacity are different constraints.
Optimizer-state sharding also differs from replacing the optimizer's mathematical update rule. A name containing optimizer should not obscure that ZeRO primarily addresses distributed storage and execution of training states.

Business use

Evaluate ZeRO against a concrete training workload. Record model size, precision, optimizer, sequence length, batch configuration and number of participating processes so memory results can be interpreted.
Compare peak memory, throughput and overall cost together. The ability to train a larger model is useful only if that model serves the project's objectives within available resources.
Include saving and restoring distributed training states in the operational evaluation. A configuration that trains successfully but cannot reliably resume an interrupted run creates a separate practical problem. Keep the configuration and compatible software versions with the experiment record.

People also ask

Does ZeRO make the model smaller?

It reduces redundant per-process storage, not necessarily the model's parameter count.

Does Stage 3 include the earlier stages?

Yes. It partitions parameters as well as the gradient and optimizer states covered earlier.

Is offloading mandatory?

No. It is a related configuration choice that can complement partitioning.

Check your understanding

Why is the Stage 1 estimate seven GB?

Answer: The twelve-GB optimizer-related share becomes three GB, while two GB each of weights and gradients remain.

What changes between Stages 2 and 3?

Answer: Parameters become partitioned too.

Why is four GB not a complete device-memory requirement?

Answer: The example excludes activations, temporary buffers and other runtime allocations.

Related terms

• Data parallelism
• Gradient
• Adam
• optimizer state
• parameter sharding
• ZeRO-Offload
• ZeRO-Infinity
• activation checkpointing

Learn next

Repeat the calculation for eight processes and a different optimizer-state size. Then compare that persistent-state estimate with an actual peak-memory profile, explaining the allocations the estimate leaves out.

More in Generative AI and language

• Unidirectional language model - A unidirectional language model predicts tokens using context from one direction in a sequence.
• Unsupervised Pretrained Network - An unsupervised pretrained network is a neural network whose parameters are first learned using an objective that does not require the eventual task's...
• Variational autoencoder - A variational autoencoder, or VAE, is a probabilistic generative model that learns a latent-variable representation of data together with an approximate...
• Vector quantization - Vector quantization represents an input vector by one of a finite set of representative vectors called a codebook.
• Vibe coding - Vibe coding is an informal term for creating software by describing desired behavior to a generative AI system and iterating on what it produces.
• Word tokenization - Word tokenization divides text into word-level units and, depending on the rules, separate punctuation or other tokens.

Sources

Rajbhandari and colleagues, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, original research abstract. https://arxiv.org/abs/1910.02054 DeepSpeed, Zero Redundancy Optimizer tutorial: stage definitions and memory accounting. https://www.deepspeed.ai/tutorials/zero/ DeepSpeed, ZeRO documentation: partitioning, parameter gathering and offload configuration. https://deepspeed.readthedocs.io/en/latest/zero3.html The four-process calculation and fictional research case are original teaching illustrations.

Previous: Zero-inflated Poisson regression Next: Zero-shot