Federated Averaging (FedAvg)

Unlike traditional distributed training that relies on a single gradient step, FedAvg allows client devices to perform multiple local epochs of training on their private data partitions before communication. Each client computes weight updates by minimizing the loss function on its local dataset. This reduces the frequency of communication rounds and is critical for handling non-IID (non-Independently and Identically Distributed) data distributions, where a client's local dataset may not statistically represent the global population.

The central server orchestrates the learning process by collecting locally-computed model updates, not raw data. The global model is updated by computing a weighted average of these client updates. The standard weighting scheme is proportional to the size of the local dataset:

• Formula: w_global = Σ (n_k / n) * w_k
• n_k: Number of samples on client k
• n: Total number of samples across all selected clients This ensures that clients with more data exert a proportionally greater influence on the global model.

In each communication round, the server randomly samples a fraction C of available clients to participate. This is a critical hyperparameter that balances convergence speed against communication overhead and straggler effects. A typical value for C is between 0.1 and 1.0. Selecting a subset of clients introduces stochasticity into the optimization process, which has been shown empirically to improve the generalization of the final model by acting as an implicit regularizer.

FedAvg is designed to minimize the communication bottleneck, which is the primary constraint in federated networks. By increasing the number of local epochs (E) and batch sizes (B), the algorithm reduces the total number of communication rounds required for convergence. This is essential for deployment on bandwidth-limited edge devices such as smartphones and IoT sensors. Techniques like gradient compression and quantization are often layered on top of FedAvg to further reduce the bitrate of uploaded model updates.

The core architectural principle of FedAvg is that raw data never leaves the client device. Only model updates (gradients or weights) are transmitted. This provides a baseline level of data minimization and privacy. However, it is not a complete privacy guarantee; model updates can leak sensitive information through gradient inversion attacks. Production systems typically combine FedAvg with formal privacy mechanisms like Differential Privacy or Secure Multi-Party Computation (SMPC) to provide provable cryptographic guarantees against such leakage.

Real-world federated networks consist of devices with vastly different compute capabilities, network connectivity, and availability. FedAvg is inherently robust to stragglers—slow or failing devices—because the server can simply drop clients that do not report updates within a deadline. The algorithm does not require synchronous updates from all clients, making it suitable for asynchronous environments. This tolerance for variable latency and hardware profiles is what distinguishes federated learning from tightly-coupled data-center distributed training.

A fundamental challenge in federated learning where the local datasets on client devices are not independent and identically distributed. This violates a key assumption of standard SGD.

• Label skew: Some clients may only have data from a subset of classes.
• Feature skew: The same label can look different across clients (e.g., different handwriting styles).
• Impact on FedAvg: Severe non-IIDness causes client-drift, where local models diverge from the global optimum, slowing convergence and degrading final model accuracy.

The strategy for choosing which subset of available devices will participate in a given training round. Random selection is standard, but advanced policies exist.

• Random sampling: A fraction C of clients is selected uniformly at random each round.
• Strafied sampling: Ensures representation across different data distributions or device types.
• Resource-aware selection: Chooses clients based on battery level, network type (Wi-Fi vs. cellular), or idle state to minimize user disruption and drop-out rates.

A primary bottleneck in FedAvg, as uploading model updates can be slow and costly on mobile networks. Techniques to compress these updates are critical for practical deployment.

• Gradient compression: Methods like sparsification (sending only the top-k largest gradient values) and quantization (reducing bit-width from 32-bit floats to 2-8 bits).
• Local computation trade-off: Increasing local epochs reduces communication rounds but can exacerbate client-drift on non-IID data.
• FedAvg's role: The algorithm's design inherently reduces communication by a factor equal to the number of local steps compared to naive distributed SGD.