Federated Averaging (FedAvg)

Instead of centralizing raw data, FedAvg pushes the computation to the edge. Each client downloads the global model and performs multiple epochs of local training using its private dataset. This local computation is the core privacy-preserving mechanism, as raw data never leaves the device. The number of local epochs is a critical hyperparameter; more local steps reduce communication but can cause client drift in Non-IID Data settings.

The central server aggregates locally trained model weights by computing a weighted average, typically proportional to the size of each client's local dataset. The formula is: w_global = Σ (n_k / n) * w_k where n_k is the number of samples on client k and n is the total samples across all selected clients. This ensures clients with more data have a proportionally greater influence on the global model.

FedAvg reduces communication overhead by allowing clients to perform multiple local SGD steps before sending updates to the server. This contrasts with Federated SGD, which requires communication after every single batch. Key trade-offs:

• More local epochs = Less communication, higher risk of client drift
• Fewer local epochs = More communication, better convergence on heterogeneous data
• Gradient compression techniques like quantization can further reduce bandwidth requirements.

In each training round, the server selects a random subset of available clients to participate. This sampling introduces stochasticity that can actually improve generalization. Critical considerations include:

• Device availability: Mobile devices may only participate when idle, charging, and on Wi-Fi
• Straggler mitigation: Setting a timeout to drop slow clients prevents bottlenecking the round
• Bias in selection: Non-uniform sampling can skew the global model toward over-represented data distributions

While FedAvg keeps raw data local, the shared model updates are not inherently private. Gradient Inversion attacks can reconstruct training samples from the gradients of shallow networks. Membership Inference attacks can determine if a specific record was used locally. This necessitates combining FedAvg with privacy-enhancing technologies like Differential Privacy (via DP-SGD) or Secure Aggregation protocols that cryptographically mask individual updates from the server.

Real-world federated data is almost always Non-IID—different clients have skewed label distributions or entirely different feature distributions. FedAvg can struggle with this heterogeneity, leading to divergent local models and a degraded global model. Mitigation strategies include:

• FedProx: Adds a proximal term to local objectives to keep updates close to the global model
• SCAFFOLD: Uses control variates to correct for client drift
• Personalized Federated Learning: Maintains local model components alongside the global model