Weighted Averaging

The defining feature of weighted averaging is that each client's update is weighted by the size of its local dataset. The formula is: Global Update = Σ (n_k / N) * Δθ_k, where n_k is the number of samples on client k, N is the total samples across selected clients, and Δθ_k is the client's model delta. This ensures clients with more data have a proportionally larger influence on the global model, aligning the optimization with the true empirical data distribution.

Weighted averaging is a primary defense against the challenges of non-IID data. By weighting updates by local dataset size, the algorithm implicitly assumes that a larger local dataset is a better statistical representation of a sub-population within the global distribution. This weighting helps anchor the global model towards a consensus that balances the varied local objectives, though it does not fully eliminate client drift without additional techniques like proximal terms or control variates.

Under standard convex optimization assumptions (e.g., smooth, bounded gradients), Federated Averaging with weighted averaging provides a formal convergence guarantee to a stationary point of the global objective. The weighting by n_k is crucial for this proof, as it makes the aggregated update an unbiased estimator of a full-batch gradient computed over the union of all client data. The convergence rate is impacted by data heterogeneity and partial client participation.

Weighted averaging enables communication efficiency by allowing multiple local epochs of training per communication round, unlike Federated SGD (FedSGD) which performs only one gradient step. This drastically reduces the number of costly communication rounds required for convergence. The weighting accounts for the varying amounts of local work (more epochs on more data) when combining updates, preventing the global model from being biased towards clients that perform more optimization steps.

Weighted averaging is compatible with secure aggregation protocols. The server can compute the weighted sum of encrypted client updates without decrypting individual contributions, preserving client privacy. The weights (n_k / N) must be known to the server or computable from encrypted metadata. This combination provides both formal privacy guarantees and correct, proportional aggregation, which is essential for sensitive applications like healthcare federated learning.

Standard weighted averaging serves as the foundation for more sophisticated federated optimization algorithms designed to improve stability and speed:

• FedProx: Adds a proximal term to the local loss, but still uses weighted averaging on the resulting updates.
• SCAFFOLD: Uses control variates to correct client updates before they are weighted and averaged.
• FedNova: Normalizes updates by the number of local steps before applying the data-size weight, correcting for workload heterogeneity.
• FedOpt Framework: Applies adaptive optimizers (e.g., FedAdam) to the sequence of weighted-averaged updates on the server.

Federated Averaging (FedAvg) is the foundational iterative algorithm that employs weighted averaging as its core aggregation mechanism. The server computes a weighted average of client model updates, where each client's weight is proportional to its local dataset size. This ensures the global objective approximates the empirical risk over the entire distributed dataset.

• Standard Protocol: The algorithm proceeds in communication rounds: server broadcasts model, clients train locally, server performs weighted averaging of updates.
• Mathematical Basis: The weighted average directly minimizes the global objective function when client data is IID, making it the theoretically optimal aggregation strategy under ideal conditions.

Statistical Heterogeneity refers to the non-independent and identically distributed (non-IID) nature of data across federated clients, which is the primary challenge for naive weighted averaging. When local data distributions diverge, a simple weighted average of updates can cause client drift, where local models optimize for their unique distributions, harming global convergence.

• Impact on Weighting: The standard dataset-size weighting becomes suboptimal, as it does not account for distributional shift.
• Algorithmic Adaptations: This challenge motivates advanced algorithms like FedProx and SCAFFOLD, which modify the local objective or use control variates to correct the drift induced by heterogeneity.

FedProx is a federated optimization algorithm designed to stabilize training under statistical heterogeneity. It modifies the local client objective function by adding a proximal term that penalizes the distance between the local model and the global model. This effectively constrains local updates, mitigating client drift.

• Relation to Weighted Averaging: The server still performs a standard weighted average of the constrained client updates. FedProx ensures these updates are better aligned before aggregation.
• μ (mu) Parameter: A hyperparameter controlling the strength of the proximal regularization. A larger μ forces clients to stay closer to the global model.

SCAFFOLD (Stochastic Controlled Averaging for Federated Learning) uses control variates—client and server correction terms—to reduce the variance in client updates caused by data heterogeneity. It corrects the local update direction before transmission to the server.

• Corrected Updates: Clients compute updates using both their local gradient and a correction term that estimates the difference between their local and global update directions.
• Aggregation Impact: The server performs weighted averaging on these variance-reduced updates, leading to significantly faster and more stable convergence compared to standard FedAvg under non-IID conditions.

These are two fundamental protocols governing when the server performs the weighted averaging step.

• Synchronous Aggregation (FedAvg Standard): The server waits for updates from all selected clients in a communication round before computing the weighted average. This is simpler and provides deterministic convergence but is vulnerable to stragglers (slow clients).
• Asynchronous Aggregation: The server updates the global model via a weighted combination immediately upon receiving an update from any client. This improves system efficiency but introduces staleness and requires careful weighting to maintain convergence, often weighting updates based on their age or client dataset size.

The FedOpt framework generalizes server-side aggregation beyond simple weighted averaging. Instead of directly applying the weighted average of updates to the global model, the server treats the aggregated update as a pseudo-gradient and applies an adaptive optimizer.

• FedAdam: The server uses the Adam optimizer (with adaptive learning rates per parameter) on the aggregated client updates. The weighted average provides the first moment (m), and a second moment (v) is tracked on the server.
• FedAvgM: Incorporates server momentum into the weighted averaging step. The update applied to the global model is a combination of the current weighted average and the previous update direction, accelerating convergence and smoothing the optimization path.