FedSGD

The defining characteristic of FedSGD is that each participating client performs exactly one step of stochastic gradient descent on its local data per communication round. This is mathematically equivalent to setting the number of local epochs to 1 and using a full batch size in the more general FedAvg algorithm. This single-step constraint simplifies convergence analysis and establishes a clear baseline for communication efficiency.

FedSGD represents the most communication-heavy end of the federated optimization spectrum. Because clients compute only a single gradient step, achieving model convergence requires a very large number of communication rounds. This characteristic makes it a useful benchmark for evaluating the communication efficiency of more advanced algorithms like FedAvg, which perform multiple local steps to amortize the cost of each round.

• High Round Count: Many rounds are needed for convergence.
• Benchmark Utility: Serves as a baseline for measuring communication reduction.

In FedSGD, clients compute and transmit the gradient of the loss function with respect to the global model parameters, rather than transmitting updated model weights. The server then performs a straightforward aggregation, typically a weighted average of these gradients based on local dataset sizes, and applies the aggregated gradient to update the global model. This process is a direct federated analogue of centralized mini-batch SGD.

FedSGD's simplicity makes it the preferred algorithm for theoretical analysis and establishing convergence guarantees. Its update rule closely mirrors centralized stochastic gradient descent, allowing researchers to extend classical optimization proofs to the federated setting by accounting for client sampling and data heterogeneity. It provides a clean framework for understanding fundamental federated learning challenges like client drift under non-IID data.

FedSGD is a special case of Federated Averaging (FedAvg). The relationship is defined by two hyperparameters:

• Local Epochs (E): FedSGD sets E = 1.
• Local Batch Size (B): FedSGD uses the full local dataset as the batch, so B = ∞.

When E > 1 or B is less than the full dataset size, the algorithm becomes FedAvg. This relationship positions FedSGD as the foundational building block from which more practical, communication-efficient variants are derived.

While theoretically important, FedSGD is rarely used in production federated systems due to its prohibitive communication costs. The constant synchronization after every single gradient step creates a severe bottleneck, especially with slow or unreliable client connections. Practical deployments almost always use FedAvg or its variants (like FedProx, SCAFFOLD) to perform multiple local computation steps, thereby drastically reducing the number of required communication rounds and improving overall system efficiency.

FedAvg is the canonical federated learning algorithm that generalizes FedSGD. While FedSGD performs a single gradient step per round, FedAvg allows clients to perform multiple local epochs of training on their data before sending an update. This drastically reduces communication frequency by performing more computation locally.

• Key Difference: FedSGD is equivalent to FedAvg with local_epochs=1 and using the full local dataset as a single batch.
• Communication-Computation Trade-off: FedAvg amortizes the cost of communication over more local computation, making it practical for networks with limited bandwidth.

Client drift is a primary optimization challenge caused by statistical heterogeneity (non-IID data) across clients. When clients perform multiple local steps (as in FedAvg), their local models can diverge significantly from the global objective, hindering convergence.

• Cause: Local optimization on biased data distributions pulls models toward local minima.
• Impact on FedSGD: FedSGD experiences less severe drift per round due to its single-step nature, but convergence is slower. Algorithms like FedProx and SCAFFOLD were developed explicitly to correct for client drift.

FedProx is a federated optimization algorithm designed to handle data heterogeneity by modifying the local client objective function. It adds a proximal term that penalizes the distance between the local model and the global model received at the start of the round.

• Mechanism: The proximal term (μ/2) * ||w - w^t||^2 acts as a regularizer, constraining local updates and directly mitigating client drift.
• Relation to FedSGD: In the limit where the proximal parameter μ is very large, FedProx forces clients to take very small steps, behaviorally resembling FedSGD's constrained update.

SCAFFOLD (Stochastic Controlled Averaging for Federated Learning) uses control variates—correction terms maintained on both server and clients—to reduce the variance in client updates. This corrects for the 'client drift' inherent in local SGD steps under non-IID data.

• Core Idea: It estimates the drift between the client's update direction and the true global direction, applying a correction.
• Comparison: While FedSGD has high communication cost and FedAvg has drift, SCAFFOLD aims for faster convergence with similar communication frequency to FedAvg but with added correction state.

Synchronous aggregation is the standard protocol used by FedSGD and FedAvg, where the server waits for updates from all selected clients in a communication round before proceeding. This creates a consistent global update but is vulnerable to stragglers (slow or offline clients).

• FedSGD Context: FedSGD is inherently synchronous; each round requires all participants to compute and return a gradient.
• Alternative: Asynchronous aggregation updates the model immediately upon receiving any client's update, which can improve system efficiency at the cost of introducing staleness and potential convergence instability.

A model delta (or parameter update) is the mathematical difference between a client's locally updated model parameters and the global model parameters it received. In FedSGD, this delta is precisely the computed gradient (or a scaled version of it) for the client's local data batch.

• Transmission: The delta, not the raw data or full model, is sent to the server for update aggregation.
• Compression: Techniques like quantization and sparsification are applied to the delta to reduce communication overhead, a field known as communication-efficient federated learning.