Does FedGD need a central server?

The graph-based update described here can operate through neighbor exchanges; other federated architectures use a server.

Will all participants learn the same model?

Not necessarily. This formulation permits different local models balanced against an agreement penalty.

Does every step decrease the objective?

No universal guarantee applies. Step size and properties of the objective matter, and nonconvex settings need particular care.

AI Glossary / Mathematics and optimization / Advanced

Federated gradient descent

Definition

Federated gradient descent, or FedGD, is a distributed training method that applies gradient updates across participants while their training data remain local. In the Aalto source's graph-based formulation, each participant maintains its own model and exchanges parameters with neighboring participants. Its update combines a gradient of local training loss with a term encouraging connected models to stay similar. This entry uses that explicitly personalized formulation.

What it means

The word federated describes where learning takes place and how participants collaborate. Gradient descent describes the numerical update. These are separate design choices: training can be distributed without requiring every participant to have an identical final model. In the graph-based setting, a participant balances fitting its own observations with agreement across selected connections. A coordinator is not required by the mathematical update. Other literature can use related terminology for a shared global model, so readers should check the objective and communication pattern before comparing algorithms by name.

How it works

For local parameters w_i, define an objective as the sum of differentiable local losses plus α times the sum over undirected edges of A_ij times the squared distance between w_i and w_j. Count each edge once. The gradient for node i is its local loss gradient plus 2α times the weighted sum of differences from neighbors. Each synchronous round exchanges current parameters, calculates these terms and subtracts a learning-rate-scaled gradient. All updates use the previous round's parameters. Full local gradients distinguish this version from a stochastic approximation using sampled observations.

Three practical examples

Several workshops can investigate separate equipment models linked according to task similarity, using neighbor information without pooling their raw maintenance records.
A distributed optimization exercise can compare independent local fitting with graph-coupled fitting while varying the coupling strength.
An engineering team can measure how synchronization delays affect the time needed to execute a prescribed number of gradient rounds, separately from the quality of the resulting predictions.

Think of it this way

Think of several people adjusting connected sliders. Each has a preferred position based on local evidence, while elastic links discourage large differences between connected sliders. They move together in small coordinated steps. The analogy explains competing influences, but real parameters may have millions of coordinates, and neither the connections nor a small movement guarantees useful predictions or confidential messages.

Worked example

Consider two participants with scalar parameters u and v. Their local losses are L_1(u) = u²/2 and L_2(v) = (v − 4)²/2. Give their single edge weight 1 and choose α = 1/2. The complete objective is F(u, v) = u²/2 + (v − 4)²/2 + (u − v)²/2. Its two gradient components are 2u − v and 2v − u − 4. Start at the independent local optima (0, 4). The gradient is (−4, 4), so a simultaneous step with learning rate 0.1 gives (0.4, 3.6). The objective falls from 8 to 0.08 + 0.08 + 5.12 = 5.28. The local losses have increased slightly, but the disagreement penalty has decreased enough to reduce the total. Solving the two zero-gradient equations gives u = 4/3 and v = 8/3. These values differ: finite coupling does not force a single shared parameter. At that solution the objective is 8/3, below the value 4 obtained by imposing u = v = 2. The example shows the importance of identifying which objective is being solved. Using the newly updated u when calculating v would define a different update schedule from this synchronous step.

Illustrative case study

In a fictional pilot, Alder Workshops compares local machine-condition models at two sites. Engineers agree on compatible features and parameter meanings before defining a collaboration edge. They log the local losses, disagreement penalty and total objective separately. A reduction in the total is followed by evaluation on each site's held-out observations, because improved agreement alone is not a business outcome. The pilot illustrates a controlled comparison and does not claim a measured accuracy improvement or a completed deployment.

Common misconceptions and mistakes

Keeping records local does not guarantee that shared parameters reveal nothing about them. FedGD is not automatically a privacy mechanism. Exact local gradients do not make optimization immune to numerical error or poor learning-rate choices. Graph connectivity does not imply that all local models will become identical under a finite penalty. A decline in the training objective does not establish better generalization. Finally, incompatible parameter meanings cannot be repaired merely by subtracting two vectors of equal length; collaboration requires a meaningful shared representation.

Business use

Define the objective before choosing infrastructure. Record local loss scaling, edge weights, coupling strength, step size and synchronization assumptions. Check whether participants can compute full local gradients within the available time and energy budgets. Monitor communication volume and unavailable participants alongside optimization diagnostics. Evaluate local performance as well as aggregate measures, especially when participants have different data distributions. Review what transmitted parameters can disclose and add appropriate protections where needed. Changing the graph or loss normalization changes the problem, so compare experiments with those choices documented.

Check your understanding

Why does the first worked step increase the local losses while decreasing the total objective?

Answer: The objective includes a disagreement penalty. Moving the parameters toward each other reduces that penalty by more than the increase in the two local losses.

Related terms

Federated learning network
Gradient descent
Generalized total variation minimization
Federated stochastic gradient descent
Synchronization
personalized federated learning

Learn next

repeat the two-node calculation with α = 0 and compare the optimum. Then increase the coupling and observe how the local parameters approach each other. Implement simultaneous updates using a saved copy of the previous parameter vector and compare them with sequential updates.

Sources

Sources: Alexander Jung, Federated Learning: From Theory to Practice, 2025 author preprint, Section 5.3 and Algorithm 5, https://arxiv.org/pdf/2505.19183 ; Jung and colleagues, The Aalto Dictionary of Machine Learning, https://aaltodictionaryofml.github.io/ADictML_Main.pdf . The two-node objective, numerical step and fictional workshop case are original. Terminology is explicitly scoped to the source's graph-based personalized formulation.

Previous: Federated averaging Next: Federated learning