Federated Averaging (FedAvg)

Unlike traditional distributed training that requires constant communication, FedAvg allows each client to perform multiple local epochs of training on its private data before communicating. This reduces communication rounds by 10–100x compared to federated SGD. The client computes weight updates by minimizing the local loss function on its own non-IID data partition, producing a partially optimized model state that captures site-specific statistical patterns before aggregation.

The central server constructs the new global model by computing a weighted arithmetic mean of the local model parameters. The weighting coefficient is typically proportional to the number of training samples each client used, ensuring that clients with larger datasets exert proportionally greater influence on the global consensus. This aggregation step is mathematically equivalent to minimizing a global loss function that is the weighted sum of local empirical risks.

In each communication round, the server selects a fraction C of available clients to participate, rather than waiting for all clients to respond. This partial participation mechanism is critical for scaling to cross-device deployments with unreliable endpoints. The selection fraction introduces a trade-off: lower C values reduce communication cost but increase variance in the aggregated update, potentially slowing convergence or introducing bias toward the selected subset.

FedAvg exhibits convergence instability when local data distributions are statistically heterogeneous. If client datasets are highly skewed—differing in label distribution, feature distribution, or concept drift—the local optima can diverge significantly. Averaging divergent weight vectors can produce a global model that performs poorly on any individual client. This phenomenon, known as weight divergence, motivates advanced variants like FedProx and SCAFFOLD that add corrective terms to local objectives.

FedAvg reduces communication by a factor proportional to the number of local epochs performed between synchronization rounds. The total communication cost per round is 2 × model size per participating client (upload and download). For large neural networks, this bandwidth requirement can become prohibitive. Compression techniques such as gradient quantization, sparsification, and structured updates are often layered on top of FedAvg to further reduce the bitrate without sacrificing convergence guarantees.

FedAvg provides a foundational privacy property: raw data never leaves the client device. Only model weight updates are transmitted. However, these weight updates are not inherently private—gradient leakage attacks can reconstruct training samples from shared parameters. Production deployments therefore combine FedAvg with differential privacy (adding calibrated noise to updates) and secure aggregation (encrypting individual contributions so the server only sees the sum) to achieve formal privacy guarantees.

A cryptographic protocol that ensures the central server in a FedAvg round can only compute the sum of model updates, not inspect individual client contributions. This prevents the server from performing gradient leakage attacks to reconstruct private training data. Secure aggregation typically relies on Shamir's secret sharing and pairwise masking, allowing the global model to benefit from all clients while mathematically guaranteeing that no single participant's update is exposed in plaintext.

The primary failure mode of vanilla FedAvg. When client data is statistically heterogeneous—meaning local distributions differ in label balance, feature distribution, or concept—simple weight averaging can lead to severe client drift and a degraded global model. Strategies to mitigate this include: - FedProx for constrained local updates - SCAFFOLD for variance reduction using control variates - Personalized Federated Learning (pFL) to forgo a single global model entirely

Standard FedAvg assumes all clients are honest. In adversarial settings, a single malicious client can submit arbitrary updates to poison the global model. Byzantine-robust aggregation replaces simple averaging with robust statistics like coordinate-wise median, geometric median, or Krum. These operators filter out outliers before aggregation, ensuring the global model remains correct even when a fraction of clients are actively malicious or have suffered catastrophic hardware failure.

FedAvg shares model weights, not raw data, but weights alone can leak information through membership inference or model inversion attacks. Differential Privacy provides a formal guarantee by adding calibrated Gaussian noise to the aggregated update before applying it to the global model. The privacy budget, controlled by the parameter epsilon (ε), quantifies the maximum information leakage risk, allowing institutions to provably bound the privacy loss incurred during each federated training round.

FedAvg behaves differently depending on the deployment topology. Cross-silo FL involves a small number (2-100) of reliable institutional clients (e.g., hospitals) that participate in every round, making synchronous FedAvg straightforward. Cross-device FL involves millions of unreliable edge devices (e.g., smartphones) with intermittent connectivity. This requires FedAvg variants that handle client dropout, stragglers, and limited upload bandwidth, often selecting only a random subset of available clients per round.