Federated Averaging (FedAvg)

FedAvg's primary characteristic is that raw training data never leaves the client device. Instead of centralizing datasets, the algorithm operates by sending only the model updates (e.g., weight deltas or gradients) to a coordinating server. This architecture is the cornerstone of its privacy promise, as sensitive user data remains locally stored. It enables training across data silos where legal, competitive, or ethical barriers prevent data pooling, such as in healthcare between hospitals or in finance between banks.

The algorithm proceeds in synchronized communication rounds. Each round consists of:

• Server Broadcast: The central server selects a subset of clients and sends the current global model.
• Local Computation: Each client performs multiple steps of Stochastic Gradient Descent (SGD) on its local data.
• Secure Aggregation: Clients send their updated model weights back to the server.
• Weighted Averaging: The server computes a new global model by averaging the client updates, typically weighted by the number of local training examples. This creates a feedback loop where the global model iteratively improves through decentralized knowledge consolidation.

FedAvg is explicitly designed to minimize the communication bottleneck, which is often the primary constraint in cross-device federated learning. It reduces the frequency of communication by performing substantial local computation (multiple SGD epochs) on each client between aggregation rounds. This trades off communication cost for increased client compute, which is favorable given that modern edge devices (smartphones, sensors) have abundant local processing power but limited or metered bandwidth. Advanced variants incorporate gradient compression, sparsification, and quantization to further reduce the size of transmitted updates.

A defining challenge FedAvg must handle is non-IID data across clients. In real-world deployments, each client's local dataset is not a representative sample of the global distribution. For example, a user's phone contains photos unique to them, or a hospital's patient demographics are local. This statistical heterogeneity causes client drift, where local models diverge, potentially harming global convergence. FedAvg's robustness to non-IID conditions is limited, leading to the development of improved algorithms like FedProx (which adds a proximal term) and SCAFFOLD (which uses control variates) to correct for this drift.

FedAvg operates under the assumption of partial participation—in each round, only a fraction of the total client population is available for training due to device availability, network connectivity, or power constraints. It must also accommodate system heterogeneity, where clients have vastly different computational capabilities (e.g., old vs. new phones), storage, and network speeds. A robust FedAvg implementation must handle stragglers and dropouts gracefully, often using asynchronous updates or timeout mechanisms to ensure progress is not blocked by the slowest participants in the federation.

While FedAvg enhances privacy by keeping data local, the model updates themselves can leak information. Without additional safeguards, a malicious server could perform model inversion or membership inference attacks on the shared gradients/weights. Therefore, FedAvg is typically combined with complementary privacy-enhancing technologies (PETs) to provide rigorous guarantees:

• Differential Privacy (DP): Adding calibrated noise to client updates before aggregation.
• Secure Multi-Party Computation (SMPC): Cryptographic protocols for secure aggregation that prevent the server from seeing individual updates.
• Homomorphic Encryption (HE): Allowing computation on encrypted updates. FedAvg provides the architectural foundation; these PETs build the verifiable privacy layer on top.

The overarching decentralized machine learning paradigm where a shared global model is trained collaboratively across multiple client devices or servers holding local data samples, without exchanging the raw data itself. FedAvg is the foundational algorithm for this paradigm.

• Core Principle: Keep training data localized.
• Key Challenge: Coordinating learning across heterogeneous, potentially unreliable devices.
• Primary Goal: Learn a global model from decentralized data silos.

A rigorous mathematical framework that provides a quantifiable privacy guarantee for individuals in a dataset. It ensures the output of a computation (e.g., a model update) is statistically indistinguishable whether any single individual's data is included or excluded.

• Applied in FL: Often used to add calibrated noise to client updates before aggregation.
• Privacy Budget (ε): A parameter controlling the strength of the guarantee; lower ε means stronger privacy.
• Relation to FedAvg: DP can be integrated into FedAvg to create a private federated averaging algorithm.

Refers to Non-Independent and Identically Distributed data, the statistical heterogeneity that is the norm, not the exception, in federated learning. Each client's local dataset has a unique distribution.

• Example: Smartphone keyboards learning from different users' typing habits.
• Challenge for FedAvg: Causes client drift, where local models diverge, impairing global convergence.
• Mitigation: Algorithms like FedProx and SCAFFOLD are designed to handle non-IID data more effectively than vanilla FedAvg.

A cryptographic protocol that allows a central server in federated learning to compute the sum (or average) of client model updates without being able to inspect any individual client's contribution.

• Privacy Enhancement: Prevents the server from performing model inversion or membership inference attacks on individual updates.
• How it works: Often uses masking techniques where clients add secret shares that cancel out upon aggregation.
• Use with FedAvg: The secure aggregation protocol is applied to the client weight updates before the averaging step.

The two primary deployment scales for federated learning, which dictate system design and algorithm choice.

• Cross-Device FL: Involves a massive number of unreliable, resource-constrained edge devices (e.g., millions of smartphones). Focuses on partial participation, communication efficiency, and handling dropouts.
• Cross-Silo FL: Involves a small number of reliable, resource-rich organizations (e.g., 10 hospitals). Focuses on data privacy between entities, higher model complexity, and full participation.
• FedAvg's Role: The core averaging logic is used in both, but system-level implementations differ drastically.

The suite of algorithms and techniques designed to efficiently train models in the federated setting, addressing core challenges beyond simple averaging.

• Key Problems Tackled: Statistical heterogeneity (non-IID data), communication costs, system constraints, and partial participation.
• Advanced Algorithms:
  • FedProx: Adds a proximal term to local loss to limit client drift.
  • SCAFFOLD: Uses control variates (variance reduction) to correct for client drift.
• Evolution from FedAvg: These algorithms are direct successors that improve upon the stability and convergence of the original FedAvg algorithm.