Federated Averaging (FedAvg)

Each client performs multiple steps of stochastic gradient descent on its local data partition before communicating. This contrasts with single-step gradient sharing by:

• Reducing communication rounds by a factor proportional to local epochs
• Allowing clients to make meaningful progress on non-IID distributions
• Trading increased local computation for decreased network bandwidth

The server does not see raw data—only the resulting weight updates after local optimization completes.

The central server computes a weighted average of client model updates to form the new global model. The aggregation rule is:

w_global = Σ (n_k / n) * w_k

Where n_k is the number of samples on client k and n is the total samples across selected clients. This weighting ensures clients with more data exert proportionally greater influence on the global model, improving statistical efficiency.

FedAvg operates in discrete synchronized rounds:

1. Server broadcasts current global model to selected clients
2. Clients train locally for E epochs on their private data
3. Clients upload only model weights (not gradients or data)
4. Server aggregates and produces the next global model

This structure decouples training from data centralization, making it suitable for bandwidth-constrained and privacy-sensitive deployments.

FedAvg is designed to operate under non-IID data distributions where client datasets differ significantly. Key mechanisms include:

• Weighted averaging that accounts for dataset size disparities
• Multiple local epochs that allow clients to adapt to local distributions
• Implicit regularization from averaging diverse client updates

However, extreme non-IID conditions can cause client drift, where local models diverge from the global optimum, slowing convergence. Variants like FedProx address this limitation.

FedAvg provides a baseline privacy guarantee through data locality:

• Raw training data never leaves the client device
• Only model weight updates are transmitted to the server
• Individual updates are ephemeral and aggregated before use

This architectural privacy is often enhanced with secure aggregation (cryptographic masking of individual updates) and differential privacy (noise injection) to protect against gradient leakage attacks that could reconstruct training samples.

In cross-device deployments with millions of clients, FedAvg uses random subset sampling:

• Only a fraction of available clients participate in each round
• Selection can be uniform random or weighted by device capability
• Stragglers (slow clients) may be dropped to maintain round timing

This sampling introduces stochasticity that can actually improve generalization while keeping communication costs manageable at massive scale.

A generalized optimization framework that extends FedAvg by introducing a proximal term to the local objective function. This term penalizes large deviations from the global model, stabilizing training when clients have heterogeneous compute resources or non-IID data. Unlike standard FedAvg, FedProx allows for partial work—clients can perform variable amounts of local computation and still contribute meaningfully to the global model, preventing stragglers from being dropped entirely.

The primary pathology that FedAvg attempts to solve, but which worsens under extreme statistical heterogeneity. Client drift occurs when local models diverge from the global optimum because each client overfits to its local data distribution. This divergence causes the simple averaging step in FedAvg to produce a suboptimal global model. Key contributing factors include:

• Non-IID data across silos
• Varying local dataset sizes
• Different numbers of local epochs

A family of techniques that reduce the communication bottleneck inherent in FedAvg's upload phase. Instead of transmitting full 32-bit floating-point gradients, clients apply quantization (reducing to 8-bit or fewer) or sparsification (sending only the top-k largest gradient elements). These methods can reduce upload sizes by 100x or more, but introduce noise that must be compensated for by error accumulation or momentum correction on the server side.

A multi-tier architecture that introduces edge aggregators between clients and the central cloud server. Instead of all clients sending updates directly to a distant server, intermediate nodes perform local aggregation first. This reduces wide-area network latency and communication costs. FedAvg runs at each tier: client-to-edge and edge-to-cloud, creating a tree-structured averaging process that scales to millions of geographically distributed devices.

A peer-to-peer alternative that eliminates the central aggregation server entirely. Instead of a star topology, nodes exchange models directly using gossip protocols or blockchain consensus. Each node averages its model with randomly selected peers, propagating updates through the network. This removes the single point of failure and trust concentration inherent in standard FedAvg, but introduces challenges in convergence guarantees and auditability.