Federated Averaging (FedAvg)

The defining feature of FedAvg is that raw training data never leaves the client device. Instead of a central dataset, the model is trained across a distributed network of clients (e.g., mobile phones, edge servers, or hospital databases). Each client computes a local model update based on its private data. Only these mathematical updates—typically gradient vectors or new model weights—are sent to the central server for aggregation. This architecture is the technical foundation for privacy-preserving machine learning, complying with regulations like GDPR and HIPAA by design.

FedAvg operates in synchronized communication rounds. Each round consists of:

• Server Broadcast: The central server sends the current global model to a subset of available clients.
• Local Computation: Each selected client performs multiple steps of Stochastic Gradient Descent (SGD) on its local data, producing an updated model.
• Secure Aggregation: Clients send their model updates back to the server.
• Weighted Averaging: The server aggregates these updates by computing a weighted average, where the weight for each client's model is often proportional to the size of its local dataset. This new average becomes the next global model. This iterative consensus mechanism allows knowledge to diffuse across the network without data pooling.

FedAvg is explicitly designed for the non-IID (Non-Independent and Identically Distributed) and unbalanced data realities of edge networks. Client data distributions are inherently different (e.g., typing habits on one phone vs. another). The algorithm accommodates:

• Partial Client Participation: Not all clients are available or selected in each communication round, simulating real-world dropouts.
• Variable Local Computation: Clients may perform a different number of local epochs based on their compute capability and battery life.
• Asynchronous Updates: Advanced variants tolerate significant stragglers. The core averaging step is robust to these variations, though they introduce convergence challenges that require careful tuning of client selection and learning rates.

A primary goal of FedAvg is to minimize costly communication between clients and the central server, which is often the bottleneck compared to local computation. It achieves this through local training epochs. Instead of sending an update after every single batch of data (as in centralized SGD), a client performs many iterations of SGD locally. This compresses a significant amount of learning into a single, infrequent communication round. The trade-off is carefully managed: too few local epochs slow convergence; too many can cause client models to diverge from each other, harming the quality of the final averaged global model.

While the basic FedAvg protocol transmits model updates in plaintext, production systems integrate it with cryptographic secure aggregation protocols. These protocols ensure the central server can compute the sum or average of client updates without being able to inspect any individual client's contribution. This protects against privacy leaks that could theoretically be reverse-engineered from a single model update. FedAvg provides the learning framework, while secure aggregation provides an essential privacy guarantee, making the system resilient even against a curious or malicious central server. This combination is critical for high-stakes applications in finance and healthcare.

FedAvg must navigate two core challenges that distinguish it from centralized training:

• Statistical Heterogeneity (Non-IID Data): Data across clients is not uniformly distributed. This can cause client drift, where local models optimize for their local data distribution, biasing the global average and potentially harming convergence. Techniques like client learning rate decay or proximal terms are used to anchor local updates closer to the global model.
• System Heterogeneity: Devices have varying availability, compute speed, and network connectivity. FedAvg's design of partial participation and tolerance for stragglers addresses this, but requires robust client sampling strategies and potentially asynchronous aggregation variants to maintain efficiency in real-world deployments.

A rigorous mathematical framework for quantifying and limiting the privacy loss incurred when an individual's data is used in a computation. In federated learning, it is often applied at the client before updates are sent or at the server during aggregation.

• Mechanism: Adds carefully calibrated noise (e.g., Gaussian or Laplacian) to the model updates or the aggregation process.
• Privacy-Accuracy Trade-off: Provides a provable (ε, δ)-privacy guarantee. Increasing privacy (lower ε) typically reduces final model accuracy due to added noise.
• FedAvg Integration: Algorithms like DP-FedAvg formally incorporate differential privacy into the federated averaging loop, making the entire training process privacy-preserving.

A property of a distributed system that enables it to reach correct consensus and function reliably even when some components fail or act maliciously (send arbitrary, incorrect data). This is a critical consideration for robust FedAvg in adversarial environments.

• Challenge in FedAvg: A malicious client could send poisoned model updates to skew the global model. Standard averaging is vulnerable to such Byzantine attacks.
• Robust Aggregation: BFT-inspired FedAvg variants replace the simple mean with robust estimators like coordinate-wise median, Krum, or trimmed mean, which can filter out a minority of malicious updates.
• Guarantee: Ensures the global model converges correctly as long as the fraction of malicious clients is below a certain threshold.

A subfield of cryptography that enables multiple parties to jointly compute a function over their private inputs while keeping those inputs concealed from each other. It provides a foundational primitive for secure federated aggregation.

• Comparison to FedAvg: FedAvg is a specific algorithm for model averaging. MPC is a general-purpose cryptographic framework that can be used to implement the secure averaging step of FedAvg.
• How it Works: Clients secretly share their model updates. Through a protocol of communication and computation on these shares, they can compute the global average without any party seeing another's raw update.
• Advantage over HE: Often more efficient than Homomorphic Encryption for the specific task of secure summation/averaging, though it requires more inter-client communication.

A consistency model for distributed systems where, given sufficient time without new updates, all replicas (or client models) will converge to the same state. This is the inherent consistency guarantee of the basic FedAvg algorithm.

• FedAvg Dynamics: Clients train locally on different data, creating temporary inconsistency (model drift). The periodic averaging step pulls models back toward a consensus point (the global model).
• Asynchronous Nature: FedAvg does not guarantee that all clients have the same model at every moment (strong consistency). It guarantees that with repeated communication rounds, they will eventually agree on a shared, improved model.
• Trade-off: Accepting eventual consistency allows for high availability and parallelism, as clients can work independently between synchronization rounds.

A method for improving model accuracy and quantifying predictive uncertainty by training multiple neural networks with different random initializations and aggregating their predictions. It is a centralized analogue to the model aggregation in FedAvg.

• Parallel to FedAvg: Both methods aggregate knowledge from multiple models. Deep Ensembles aggregate predictions from models trained on the same dataset. FedAvg aggregates parameters from models trained on different, partitioned datasets.
• Uncertainty Quantification: A key benefit of ensembles. Similarly, the variance of client models in FedAvg before aggregation can be analyzed to understand data heterogeneity or client drift.
• Mechanism Difference: Ensembles typically use model averaging at prediction time. FedAvg uses parameter averaging during training to create a single, unified model.