Is one sampled gradient expected to equal the full gradient?

No. Unbiasedness concerns an average over the sampling randomness, not equality for each draw.

Does a larger batch always give a better final model?

No. It can reduce sampling variability under appropriate conditions, but final performance also depends on optimization, data and resource choices.

Does this method require identical participant models?

The graph-based formulation here allows personalized local parameters coupled through selected relationships.

AI Glossary / Mathematics and optimization / Advanced

Federated stochastic gradient descent

Definition

Federated stochastic gradient descent, or FedSGD, is a federated training method that uses stochastic or approximate local gradients rather than requiring an exact gradient over every local observation at each update. In the Aalto source's graph-based formulation, participants maintain local models, exchange parameters with neighbors and combine gradient estimates with a model-agreement term. This entry explicitly follows that formulation; the name is also used in other federated settings.

What it means

Stochastic means that an update can depend on randomly selected observations or another random gradient estimate. The randomness can reduce the local work required for one step, but it introduces variation into the optimization path. Federated means that participants collaborate while their raw datasets remain local. Neither word requires identical local models or a central coordinator. When comparing implementations, identify whether they learn a shared model or personalized models, whether randomness comes from local batches or participant selection, and what information is exchanged.

How it works

For a graph-coupled differentiable objective, each participant receives its neighbors' current parameters and obtains an estimate of its own local loss gradient. It adds the derivative of the agreement penalty and takes a learning-rate-scaled step. A uniformly sampled observation can give an unbiased estimate of an average local gradient when the scaling is correct. Unbiased means that the estimate's expectation equals the target gradient at the given parameters; individual estimates can differ substantially. The Aalto method leaves the graph-coupling calculation explicit while approximating the local data-dependent gradient.

Three practical examples

A participant with a large local dataset can sample a small batch for each training step instead of scanning all local records every round.
A learning-methods study can compare exact local gradients with sampled gradients under the same graph and objective to isolate the effect of sampling.
An engineering team can investigate how batch size affects local computation time and gradient variability while separately recording the communication required by the protocol.

Think of it this way

Think of several analysts adjusting local forecasts after reviewing a randomly chosen portion of their records and comparing notes with selected colleagues. Reviewing fewer records can be faster, but each review gives a less stable picture. The analogy does not imply that any convenient subset is representative. The mathematical sampling rule and scaling determine whether the resulting estimate targets the intended gradient.

Worked example

Consider two participants with scalar parameters u and v. Participant A has targets 0 and 2; participant B has targets 4 and 6. For each observation y, use loss (w − y)²/2, and define a participant's local loss as the average of its two observation losses. The exact local gradients are u − 1 and v − 5. Add a single-edge agreement penalty (u − v)²/2. The total gradients are therefore (u − 1) + (u − v) and (v − 5) + (v − u). Start with u = v = 0. The agreement terms are zero, and an exact gradient step with learning rate 0.1 produces (0.1, 0.5). Instead, independently sample one target uniformly at each participant. If A samples 0 and B samples 6, their estimated local gradients are 0 and −6, producing (0, 0.6). Across the four equally likely target pairs, A's new parameter is either 0 or 0.2 and B's is either 0.4 or 0.6. Their expected values are 0.1 and 0.5, matching the exact step at this starting point. This one-step equality of expectations does not imply that each stochastic path follows the exact trajectory. Nor does it promise that the objective decreases after every realized update.

Illustrative case study

In a fictional experiment, Maple Devices trains local calibration models on a fixed collaboration graph. Engineers compare one-observation sampling with larger local batches while holding the loss definition and coupling constant. They repeat trials with documented random seeds and assess both optimization variability and held-out prediction quality. A shorter local calculation is evaluated alongside the number of rounds needed to reach a chosen criterion. The example illustrates experimental design, not a claim that smaller batches always reduce total training cost.

Common misconceptions and mistakes

A random-looking subset is not necessarily an unbiased sample. Selecting only recent or easy records can change the target unless the objective or weighting accounts for that choice. Unbiased gradient estimates do not guarantee identical model outcomes across runs. Reducing local batch size does not automatically reduce the size of a transmitted parameter vector. Stochastic gradients are not a privacy guarantee. Also, FedSGD terminology is overloaded: some shared-model literature emphasizes aggregation across selected clients, so this graph-based personalized update should not be silently substituted for a different algorithm with the same label.

Business use

Write down the local objective, sampling distribution, batch scaling and graph penalty before implementing updates. Check a small case where the exact gradient can be computed and compare it with the average of many sampled estimates. Monitor local workload, gradient variability, communication and per-participant performance. Record random seeds where useful for debugging, while recognizing that infrastructure can add other sources of nondeterminism. Review learning-rate schedules and stopping criteria for noisy optimization. If participant availability is also random, assess that selection mechanism separately from local observation sampling.

Check your understanding

Why is A's expected first update 0.1 even though its realized update is either 0 or 0.2?

Answer: The two targets are sampled with equal probability, so the expected updated parameter is (0 + 0.2)/2 = 0.1.

Related terms

Learn next

enumerate the four possible first-step outcomes in the worked example and verify their average. Then sample A's targets with unequal probabilities and determine when an unweighted estimate becomes biased for the original equally weighted loss. Keep the local sampling rule distinct from the rule choosing participating devices.

Sources

Sources: Alexander Jung, Federated Learning: From Theory to Practice, 2025 author preprint, Section 5.4 and Algorithms 6–7, https://arxiv.org/pdf/2505.19183 ; Jung and colleagues, The Aalto Dictionary of Machine Learning, https://aaltodictionaryofml.github.io/ADictML_Main.pdf . The two-participant sampling example and fictional calibration experiment are original. The entry uses the source's graph-based personalized FedSGD meaning explicitly.

Previous: Federated learning system Next: Federation (information technology)