Is all-reduce the same as all-gather?

No. All-reduce combines contributions with an operation; all-gather distributes the separate contributions to all participants.

Does collective mean all computers everywhere?

No. It means the processes in the specified group or communicator.

Can collective operations run asynchronously?

Nonblocking variants can overlap communication with other work, subject to the library's completion and buffer rules.

AI Glossary / Data engineering and architecture / Intermediate

Collective operation

Definition

A collective operation is a communication operation involving a defined group of processes in a parallel program. Instead of sending individual messages manually between every pair, the program requests a group action such as broadcasting data, gathering results or combining values. The communication library coordinates the required exchanges according to the operation's semantics.

What it means

A group might contain the workers participating in one distributed calculation. A broadcast distributes one process's data to the group; a gather collects contributions at one process; an all-reduce combines contributions and gives the result to every participant. The exact operation determines who supplies data and who receives output. Collective does not mean that every process in the entire machine participates: the relevant communicator or group defines the scope.

How it works

Processes invoke compatible collective calls on the agreed communication context. The library can choose an efficient communication pattern based on message size, network and implementation. A reduction applies an operation such as summation to contributions. Blocking and nonblocking variants have different completion rules, and a collective is not automatically a global barrier. Programs must respect ordering, buffer use and participation requirements. Mismatched calls or missing participants can cause errors or prevent progress.

Three practical examples

A distributed training job uses an all-reduce to combine gradients across workers before a synchronized update.
A simulation broadcasts shared configuration from one process to the rest of its group.
A parallel analytics job reduces local record counts into a total, then returns that total to one process or all processes depending on the chosen operation.

Think of it this way

Think of a meeting where everyone contributes a number and a coordinator announces the sum. The group action is clearer than asking each participant to negotiate separate conversations. The analogy is limited because a collective library may use a tree or ring without one physical coordinator, and communication completion has precise rules that differ from everyone finishing a meeting.

Worked example

Four workers count records in separate partitions and obtain 12, 18, 9 and 21. A sum reduction produces 60 at the designated receiving process. A sum all-reduce produces 60 at every participating worker. A gather instead collects the four values [12, 18, 9, 21] without summing them. These operations answer different questions even though they involve the same group. If the application needs an average, it must also know the relevant count or weights; summing local averages is not generally the same as averaging all underlying records.

Illustrative case study

Fictional case: Birch Models adds a worker to a distributed experiment, but one code path skips the collective used by the other workers. The run stops making progress. The team reviews control flow so all required participants invoke matching operations and adds clear diagnostics around the communication stage. It also checks that the group contains the intended workers. This fictional case illustrates coordination requirements rather than an issue unique to any one library.

Common misconceptions and mistakes

Do not assume every collective synchronizes all processes like a barrier. Do not modify a nonblocking operation's buffer before its completion rules permit it. Floating-point reduction order can affect the final low-order bits because addition is not perfectly associative in finite precision. A collective may move large amounts of data efficiently, but it still consumes network and memory bandwidth. Adding workers can increase communication cost even when local computation decreases.

Business use

Businesses using distributed analytics or AI should understand collectives as a possible scaling bottleneck and correctness boundary. Measure communication time separately from computation and data loading. Compare the benefit of more workers with their coordination cost. For reproducible numerical work, document precision and reduction behavior where it matters. Use established libraries and their supported semantics rather than assuming a group operation is just a convenient shorthand with no operational consequences.

Check your understanding

Every worker needs the total number of records, not the list of per-worker counts. Which conceptual operation fits?

Answer: A sum all-reduce, because it combines the counts and returns the total to every participant. A gather would provide a different result at a designated process.

Related terms

Broadcast (parallel pattern)
Reduce (pyramid operator)
Barrier (computer science)
broadcast
reduce
all-reduce
all-gather
communicator
barrier
distributed training

Learn next

Trace broadcast, gather, reduce and all-reduce for four workers with small labeled values. Mark each process's input and output. Then inspect the chosen library's rules for call ordering and completion before attempting to overlap communication with computation.

Sources

Primary standard reference: MPI Forum, MPI standard documentation, https://www.mpi-forum.org/docs/ . Official standard search excerpts consulted; full version-specific semantics and editorial review remain pending. Examples describe conceptual collective operations rather than a complete MPI program.

Previous: Collapsed Gibbs sampling Next: Collider