Federated Averaging (FedAvg)

The core mechanism of FedAvg. Instead of computing a single gradient step on a global batch, each selected client performs multiple epochs of local training on its private dataset. This involves running a standard optimizer like SGD or Adam locally. The server then averages the resulting model weight deltas (or the final weights), not the gradients themselves. This reduces the number of communication rounds required for convergence by a factor of 10–100x compared to naive gradient averaging.

In each communication round, the central server randomly selects only a fraction of available clients (e.g., 10–100 out of thousands) to participate. This is a practical necessity to manage stragglers and network load. The statistical implication is that the global update is a stochastic approximation of the true full-federation average. The subsampling rate is a critical hyperparameter that balances convergence speed against communication cost and client availability.

A defining challenge for FedAvg is that client data distributions are almost never independent and identically distributed (IID). A client's local dataset may represent a skewed subset of the global population. FedAvg's local SGD steps can cause client drift, where local models diverge from the global optimum. While FedAvg is empirically robust to moderate non-IIDness, severe skew can degrade accuracy or cause divergence, motivating extensions like FedProx and SCAFFOLD.

The server-side averaging is typically a weighted average, where each client's contribution is scaled by the size of its local dataset (n_k / n). This ensures that clients with more data have a proportionally larger influence on the global model. The formula is: w_{t+1} = Σ (n_k / n) * w^k_{t+1}. In the simplest case with equal dataset sizes, this reduces to a simple arithmetic mean. This weighting is a maximum likelihood estimator for the global objective under a uniform prior.

FedAvg's primary advantage is a dramatic reduction in communication rounds. By performing multiple local epochs (E) with a given batch size (B), clients do significant computation before transmitting. Key parameters controlling this trade-off:

• E (Local Epochs): More epochs reduce communication but increase the risk of client drift on non-IID data.
• B (Batch Size): Larger batches provide more stable local updates.
• Fraction C: The fraction of clients selected per round. This computation-to-communication ratio is the central tuning knob for practical deployments.

FedAvg provides a baseline of data minimization by design: raw training data never leaves the client device. Only model updates (weights or deltas) are transmitted. However, these updates are not inherently private. Gradient leakage attacks can reconstruct training data from the updates alone. Therefore, FedAvg is almost always deployed in conjunction with a Secure Aggregation Protocol to hide individual updates from the server and/or Differential Privacy mechanisms to add calibrated noise, providing formal privacy guarantees.

A cryptographic protocol that allows the central server to compute the sum of model updates without inspecting any individual client's contribution. In a standard FedAvg round, the server would see each client's update; SecAgg ensures the server only learns the aggregate average, mathematically preventing gradient leakage. This is the primary privacy layer deployed on top of FedAvg in production systems.

The primary privacy threat that FedAvg faces without secure aggregation. An adversary with access to individual model updates can apply gradient inversion attacks to reconstruct training data:

• Deep Leakage from Gradients (DLG): Iteratively optimizes dummy inputs to match observed gradients
• Inverting Gradients: Exploits the fully connected layer to recover pixel-perfect images
• Text Reconstruction: Recovers token sequences from language model gradients

This is why FedAvg is almost always paired with SecAgg or differential privacy in practice.

A critical property for real-world FedAvg deployments where mobile devices or edge nodes frequently disconnect. A dropout-resilient protocol ensures the server can still compute the correct aggregate even when a subset of clients fails to complete the round. Techniques include:

• Secret sharing with threshold recovery: Only a subset of shares needed to reconstruct masks
• Timeout-based round advancement: Proceed without stragglers after a deadline
• Oversampling: Select more clients than needed, anticipating some dropout

Without this, a single disconnected phone could stall the entire training round.

The core cryptographic technique enabling efficient secure aggregation. Each pair of clients agrees on a shared secret mask that is added to one client's update and subtracted from the other's. When the server sums all updates, these pairwise masks cancel out mathematically, revealing only the true aggregate. This is combined with Diffie-Hellman key agreement to establish the shared secrets and pseudorandom generators (PRGs) to expand short seeds into full mask vectors matching the model dimensions.

Techniques to prevent slow clients from becoming the bottleneck in synchronous FedAvg rounds. The global model update must wait for all selected clients, so a single slow device degrades wall-clock time. Mitigation strategies include:

• Coded computation: Redundant encoding allows recovery from a subset of responses
• Asynchronous FedAvg: Server updates the global model immediately upon receiving any update, without waiting for the full cohort
• Client selection based on hardware capability: Prefer devices with sufficient compute and battery

This is essential for cross-device FL with millions of heterogeneous phones.

A robustness technique applied before aggregation to bound the L2 norm of each client's model update to a maximum threshold. This serves dual purposes:

• Differential privacy: Clipping bounds the sensitivity of each update, enabling meaningful privacy guarantees when noise is added
• Byzantine resilience: Limits the influence of any single malicious or outlier client on the global model

In FedAvg, clipping is typically applied as: update = update * min(1, C / ||update||₂) where C is the clipping threshold. This prevents a single corrupted update from dominating the average.