Update Aggregation

The standard aggregation method where client updates are combined proportionally to the size of their local datasets. This ensures the global objective function approximates the empirical risk over the union of all client data. The formula is typically: w_global = Σ (n_k / N) * w_k, where n_k is the number of samples on client k, N is the total samples, and w_k is the client's model update. This weighting aligns the optimization with the overall data distribution, preventing clients with small datasets from being overshadowed.

A primary goal of advanced aggregation is to counteract client drift, the divergence of local models due to optimization on statistically heterogeneous (non-IID) data. Algorithms achieve this by modifying the aggregation logic:

• FedProx: Adds a proximal term to the local client loss, penalizing updates that stray too far from the global model.
• SCAFFOLD: Uses control variates (correction terms) to reduce the variance between client updates.
• FedNova: Normalizes updates by the number of local steps, ensuring clients that train more don't disproportionately influence the global model.

Aggregation is governed by a coordination protocol that defines when the server acts.

• Synchronous Aggregation (e.g., FedAvg): The server waits for updates from all selected clients in a round before aggregating. This is stable but vulnerable to stragglers (slow clients) that bottleneck the system.
• Asynchronous Aggregation: The server updates the global model immediately upon receiving any client update. This improves system efficiency and handles stragglers naturally but can introduce stability challenges, as the model is updated based on potentially stale global parameters.

Moving beyond simple averaging, the FedOpt framework treats aggregated client updates as pseudo-gradients and applies adaptive optimizers on the server. This can significantly accelerate convergence.

• FedAdam: Applies the Adam optimizer (adaptive moment estimation) to the aggregated updates.
• FedAvgM: Incorporates a momentum term on the server, using a weighted sum of past updates to smooth the optimization path. These methods add computational overhead on the server but can reduce the total number of communication rounds required.

The effectiveness of aggregation is intrinsically linked to client selection. The chosen subset of clients determines the statistical properties of the updates being combined.

• Partial Client Participation: In large-scale systems, only a fraction of clients participate per round. Aggregation must be robust to this sampling variance.
• Strategic Selection: Algorithms may select clients based on criteria like dataset size, resource capability, or data distribution to improve aggregation quality or speed up convergence. Poor selection can bias the global model.

Aggregation is the linchpin for privacy-preserving techniques. By design, it obscures individual data points, but additional cryptographic and statistical methods are layered on top for formal guarantees.

• Secure Aggregation: Cryptographic protocols (e.g., using secret sharing) allow the server to compute the sum of client updates without being able to inspect any single update.
• Differential Privacy: Noise can be added to the updates before aggregation or to the aggregate itself, providing a mathematical guarantee that the output does not reveal whether any individual's data was included in the training set.

The most fundamental challenge is that client data is rarely Independent and Identically Distributed (IID). Real-world data is statistically heterogeneous, meaning each client's local dataset has a unique distribution. This causes client drift, where local models diverge from the global objective. Simple averaging of these divergent updates can lead to a global model that performs poorly for all clients, slowing convergence and reducing final accuracy. Algorithms like FedProx and SCAFFOLD are specifically designed to mitigate this issue.

Clients vary drastically in their hardware (CPU, GPU, memory), network connectivity, and availability. This system heterogeneity creates stragglers—clients that are orders of magnitude slower than others. In synchronous aggregation, the server must wait for all selected clients, causing the entire round to bottleneck on the slowest device. This leads to severe inefficiency and wasted compute on faster clients. Mitigation strategies include:

• Asynchronous aggregation protocols.
• Deadline-based client selection, where only updates received within a time window are aggregated.
• Tiered training based on device capability.

Transmitting full model updates (e.g., millions of parameters) from many clients to a central server is a major bandwidth and latency constraint. This communication cost often dominates the total training time. Update compression techniques are critical to make federated learning feasible, especially over mobile or satellite networks. Key methods include:

• Quantization: Reducing the numerical precision of update values (e.g., from 32-bit to 8-bit).
• Sparsification: Only sending the largest or most significant gradient values, zeroing out the rest.
• Subsampling: Transmitting only a random subset of model parameters per round.

The aggregation server is a trusted entity, but individual updates can leak sensitive information about a client's private data through techniques like model inversion or membership inference attacks. Malicious clients can also perform data poisoning by submitting crafted updates to corrupt the global model. Key challenges include:

• Ensuring update privacy without seeing raw data. Solutions: Secure Aggregation protocols (e.g., using cryptographic multiparty computation) and Differential Privacy (adding calibrated noise to updates).
• Defending against Byzantine failures from adversarial clients. Solutions: Robust aggregation rules like coordinate-wise median or Krum, which are less sensitive to outlier updates.

In real-world deployments, partial client participation is the norm, not the exception. In any given communication round, only a fraction of the total client pool is available due to devices being offline, in low-power mode, or opting out. This creates a biased sampling problem, as the aggregated update only represents a non-random subset of the overall data distribution. Furthermore, clients may drop out mid-round after downloading the model but before sending an update. Aggregation algorithms must be robust to this missing data and variable participation to ensure stable convergence.

Standard weighted averaging weights updates by the size of each client's local dataset. This can lead to representation bias, where clients with large datasets dominate the global model, potentially degrading performance for clients with smaller but equally important data (e.g., rare medical conditions). The challenge is to aggregate updates in a way that produces a model that is performant, fair, and equitable across all participating data distributions. Research explores alternative weighting schemes and personalized federated learning (e.g., FedPer, FedRep) as solutions, where a shared base model is aggregated, but final layers are personalized locally.

Federated Averaging (FedAvg) is the foundational iterative algorithm where a central server coordinates multiple clients. The server periodically broadcasts the global model, selected clients perform local training over several epochs, and the server aggregates their parameter updates via a weighted average to form a new global model. It is the canonical reference for all update aggregation techniques.

Weighted averaging is the standard mathematical operation at the heart of most aggregation methods. Client updates are combined proportionally to the size of their local datasets. The formula is: global_update = Σ (n_k / N) * delta_k, where n_k is client k's data samples, N is the total samples, and delta_k is the client's model delta. This ensures the global objective aligns with the overall data distribution.

These are two fundamental communication protocols for aggregation:

• Synchronous Aggregation: The server waits for updates from all selected clients in a round before aggregating. This is standard in FedAvg but can be bottlenecked by stragglers.
• Asynchronous Aggregation: The server updates the global model immediately upon receiving any client update. This improves throughput but can lead to stale updates and convergence instability due to a lack of a consistent global state.

The FedOpt framework generalizes server-side aggregation by applying adaptive optimization algorithms (like Adam, Yogi, or Adagrad) to the averaged client updates, instead of using a simple weighted average. For example, FedAdam uses the Adam optimizer on the server, which can accelerate convergence and adapt to heterogeneous client landscapes by maintaining per-parameter momentum and variance estimates.

Client drift is the core challenge where local models diverge from the global objective due to non-IID data. Mitigation algorithms modify the aggregation process or client objective:

• FedProx: Adds a proximal term to the local loss, penalizing updates too far from the global model.
• SCAFFOLD: Uses control variates (correction terms) to reduce variance in client updates.
• FedNova: Normalizes client updates based on their local step counts before averaging to account for workload heterogeneity.

Secure Aggregation is a cryptographic protocol that allows the server to compute the sum of client updates without learning any individual client's contribution. This provides an additional layer of privacy beyond data localization. Protocols typically use multi-party computation (MPC) or homomorphic encryption to mask individual updates so that only the aggregated result can be decrypted by the server.