Federated Averaging (FedAvg)

Unlike classic distributed training that computes a single gradient on a large batch, FedAvg allows each client to perform multiple local epochs of Stochastic Gradient Descent on its private dataset. This reduces the frequency of communication rounds. The local update rule is typically: w_{t+1}^k = w_t - η ∇F_k(w_t), where w_t is the global model, η is the learning rate, and F_k is the local objective on client k.

The central server performs a coordinate-wise weighted average of the model updates received from participating clients. The aggregation is typically weighted by the number of training samples n_k on each client: w_{t+1} = Σ (n_k / n) * w_{t+1}^k. This ensures that clients with more data have a proportionally greater influence on the global model, preventing a minority of small datasets from skewing convergence.

FedAvg is designed to be communication-bound, not computation-bound. By increasing the number of local training epochs (E) and adjusting the local batch size (B), the algorithm trades increased local computation for significantly fewer communication rounds. This is critical for edge devices with expensive or slow network connections.

A defining challenge for FedAvg is statistical heterogeneity, where local datasets are non-Independent and Identically Distributed (non-IID). Client data distributions P_k(x,y) can vary wildly. While standard FedAvg can diverge under severe non-IID conditions, its core mechanism of averaging local updates provides a foundational level of robustness that more advanced algorithms like FedProx and SCAFFOLD build upon.

FedAvg provides a baseline layer of data minimization by ensuring raw data never leaves the client device. Only model updates (gradients or weights) are transmitted. However, these updates are not inherently cryptographically private; they can leak information through gradient inversion attacks. Production systems typically layer FedAvg with Differential Privacy or Secure Multi-Party Computation (SMPC) to achieve formal privacy guarantees.

In each communication round, the server selects a random fraction C of available clients to participate. This client sampling mechanism handles the reality of unreliable edge devices that may be offline, charging, or on metered connections. The fraction C is a critical hyperparameter that balances global model convergence speed against the practical availability of the client fleet.

Google's Gboard uses FedAvg to train next-word prediction and query suggestion models directly on user devices. The on-device model trains on local typing history, and only encrypted weight updates are sent to the aggregation server.

• Scale: Billions of devices participate in nightly training rounds
• Privacy: Raw keystrokes never leave the device
• Optimization: Uses Federated Stochastic Gradient Descent with adaptive clipping to bound update norms

Hospitals train diagnostic imaging models collaboratively without sharing patient data. Each institution trains locally on its PACS (Picture Archiving and Communication System) data, and only model deltas are aggregated.

• Use Case: Tumor segmentation across geographically dispersed radiology departments
• Compliance: Satisfies HIPAA and GDPR data residency requirements
• Framework: Often implemented via NVIDIA FLARE or OpenFL with FedAvg as the core aggregation algorithm

Banks and payment processors deploy FedAvg to train anomaly detection models across competing institutions. Each bank's transaction data remains on-premises while the global model learns from diverse fraud patterns.

• Architecture: Local autoencoders trained on institution-specific transaction distributions
• Aggregation: Weighted averaging proportional to local dataset size
• Benefit: Detects cross-institutional fraud rings without exposing customer transaction histories

Automotive manufacturers use FedAvg to aggregate driving model improvements from vehicle fleets. Each vehicle trains on local sensor data from edge cases encountered during operation.

• Data: Camera, LiDAR, and radar inputs processed via on-board neural processing units
• Communication: Updates transmitted over 5G or WiFi when vehicles are parked and charging
• Challenge: Handling non-IID data distributions across different geographic regions and driving conditions

Manufacturing plants deploy FedAvg to train predictive maintenance models across factory floors without centralizing proprietary operational data. Each machine's sensor telemetry trains a local model.

• Objective: Predict bearing failures and motor degradation
• Privacy: Proprietary production schedules and throughput data never leave the factory
• Aggregation Strategy: FedProx variant used to handle heterogeneous compute resources across edge devices