Federated Averaging (FedAvg)

FedAvg operates in synchronized communication rounds. In each round, a subset of clients receives the current global model, performs Local Stochastic Gradient Descent (Local SGD) for multiple epochs on their private data, and sends the resulting model update (or the full model) back to the server. The server then computes a weighted average of these updates, typically weighted by the number of training samples on each client, to produce a new global model. This iterative averaging approximates the gradient descent that would occur on a centralized dataset.

A fundamental challenge FedAvg addresses is non-IID (Independent and Identically Distributed) data across clients. Real-world device data is inherently heterogeneous (e.g., different user typing habits, local photo libraries). FedAvg's robustness to this stems from performing multiple local update steps, allowing models to partially adapt to local distributions before aggregation. However, this can lead to client drift, where local models diverge from the global objective. Advanced variants like FedProx and SCAFFOLD introduce mechanisms to explicitly correct for this drift.

In practical deployments, it is infeasible and inefficient to involve all clients in every training round due to constraints like device availability, network connectivity, and battery life. FedAvg is designed for partial client participation, where the server samples a fraction of the total client population (e.g., 1-10%) in each round. This sampling is often probabilistic, sometimes weighted by client data volume. This characteristic is crucial for scalability and mirrors the real-world intermittency of edge devices.

The primary bottleneck in federated learning is often communication bandwidth, not computation. FedAvg is explicitly designed for communication efficiency by performing substantial local computation (many SGD steps) between each communication round. This reduces the total number of rounds required for convergence compared to sending gradients after every single batch. Further efficiency is achieved through techniques like gradient compression, quantization, and top-k sparsification, which can be layered on top of the core FedAvg protocol.

FedAvg cleanly separates the optimization processes on the server and clients. The client's role is purely local model training via SGD. The server's role is purely aggregation via a simple weighted average. This decoupling allows for significant flexibility and innovation on both sides. For instance, the FedOpt framework generalizes the server's aggregation step to use adaptive optimizers like FedAdam or FedYogi instead of simple averaging. Similarly, clients can employ personalized techniques or different local optimizers.

FedAvg provides a foundational privacy-by-architecture benefit because raw training data never leaves the client device; only model updates are shared. However, these updates can potentially leak information about the underlying data. Therefore, FedAvg is typically combined with formal privacy-enhancing technologies (PETs) to provide rigorous guarantees. The most common augmentations are:

• Secure Aggregation: Cryptographic protocols that allow the server to compute the sum/average of client updates without inspecting any individual update.
• Differential Privacy: Adding calibrated noise to client updates before they are sent, providing a mathematical guarantee that the output does not reveal whether any individual's data was used in training.

FedProx is a federated optimization algorithm designed to handle statistical and systems heterogeneity. It modifies the local client's objective function by adding a proximal term that penalizes the distance between the local model and the current global model. This constraint prevents client drift, stabilizes training, and improves convergence when clients have varying computational capabilities or non-IID data distributions. It is a direct, robust enhancement to the standard FedAvg procedure.

SCAFFOLD (Stochastic Controlled Averaging) tackles the fundamental problem of client drift caused by data heterogeneity. It introduces control variates—correction terms stored on both the server and each client. These variates estimate the difference between the client's local gradient and the global gradient direction. By applying this correction during local training, SCAFFOLD achieves significantly faster convergence and better final accuracy than FedAvg on highly non-IID data, albeit with increased communication cost for the control states.

The FedOpt framework generalizes the server-side aggregation step of FedAvg. Instead of a simple weighted average of client updates, FedOpt applies an adaptive optimizer (like Adam, Yogi, or Adagrad) to the aggregated client gradients on the server. This allows the global model update to incorporate momentum, per-parameter learning rates, and other advanced optimization dynamics. FedAdam, FedYogi, and FedAdagrad are specific instantiations of this framework, often leading to improved performance on complex, non-convex models.

Local Stochastic Gradient Descent (Local SGD) is the core training procedure executed by each client in a federated learning round. Instead of performing a single gradient step, each selected client runs multiple epochs of SGD on its local dataset. This communication-computation trade-off is central to FedAvg's efficiency: it reduces the frequency of communication (saving bandwidth) at the cost of potential client drift. The number of local steps is a critical hyperparameter balancing convergence speed and final model quality.

Gradient compression is a suite of techniques to reduce the communication bottleneck in federated learning. Instead of sending full-precision model updates, clients compress their gradients before transmission. Key methods include:

• Quantization: Mapping 32-bit floats to lower-bit representations (e.g., 8-bit).
• Sparsification: Transmitting only the most significant gradient values (e.g., Top-k Sparsification).
• Error Feedback: A crucial companion technique that accumulates compression error locally and adds it to the next gradient, preserving convergence guarantees despite the lossy compression.

Asynchronous Federated Optimization departs from the synchronized round structure of FedAvg. In this paradigm, the central server updates the global model immediately upon receiving an update from any client, without waiting for a fixed cohort. Algorithms like FedAsync handle stale updates from slow clients by applying a mixing hyperparameter that decays with the update's age. This approach improves system efficiency in highly heterogeneous environments where device availability and connectivity vary dramatically, at the cost of more complex convergence analysis.