Federated Averaging (FedAvg)

The core mechanism of FedAvg is a coordinate-wise weighted average of model parameters. After local training, each client $k$ sends its updated weights $w_k$ to the server. The global model is computed as $w_{global} = \sum_{k=1}^{K} \frac{n_k}{n} w_k$, where $n_k$ is the number of local samples and $n$ is the total samples across all participating clients. This ensures that clients with larger clinical datasets exert proportionally greater influence on the global model, reflecting statistical confidence. The process is mathematically equivalent to Stochastic Gradient Descent (SGD) executed in a distributed fashion, but with the critical privacy-preserving property that raw data never leaves its origin.

FedAvg decouples local computation from global synchronization. Instead of communicating gradients after every batch, clients perform multiple local epochs of SGD on their private data before communicating. This reduces communication rounds by orders of magnitude compared to classic distributed SGD. The algorithm introduces a hyperparameter $E$, the number of local epochs, which creates a trade-off: higher $E$ reduces communication costs but increases the risk of client drift, where local models diverge toward local optima. In healthcare settings with non-IID data distributions across hospitals, this drift can slow convergence or degrade final model quality.

In realistic cross-device deployments, it is impractical to wait for thousands of edge devices to complete training simultaneously. FedAvg introduces a fractional participation parameter $C$, where only a random subset of available clients is selected for each training round. This straggler-resilient design ensures the system does not block on the slowest device. However, biased client selection can introduce statistical skew into the global model. In cross-silo healthcare settings with a small number of reliable institutional clients, $C$ is typically set to 1.0, enabling full participation and deterministic aggregation.

The original FedAvg paper proved empirical convergence on IID data partitions, but real-world clinical data is notoriously non-IID. Subsequent theoretical analysis established that FedAvg converges to a global optimum at a rate of $O(1/\sqrt{nKE})$ for strongly convex functions, even under heterogeneous data distributions, provided the learning rate is sufficiently small. However, when data distributions diverge significantly—such as one hospital specializing in rare diseases while another handles routine care—the global model can exhibit objective inconsistency, converging to a solution far from any local optimum. This motivates advanced variants like FedProx and SCAFFOLD.

FedAvg provides architectural privacy by design: raw patient data never leaves the local institution. Only model updates—gradients or weights—are transmitted. However, this does not constitute a formal privacy guarantee. Research has demonstrated that gradient leakage attacks can reconstruct training images from shared gradients, and membership inference attacks can determine if a specific patient's record was used in training. In regulated healthcare environments, FedAvg must be augmented with Differential Privacy (DP-FedAvg) or Secure Aggregation (SecAgg) protocols to provide cryptographically sound privacy assurances that satisfy HIPAA and GDPR requirements.

The primary operational constraint in FedAvg is not local compute but communication efficiency. Each global synchronization round requires transmitting full model weights—potentially millions of parameters—over hospital networks. For large models like ResNet-152 or transformer architectures, this can consume gigabytes per round. Practical deployments employ gradient compression techniques such as quantization (QSGD) and sparsification (Top-K) to reduce bandwidth by 100-1000x. The number of communication rounds required for convergence typically ranges from hundreds to thousands, making this the dominant factor in total training wall-clock time for federated healthcare systems.

A direct evolution of FedAvg that adds a proximal term to local objective functions. This penalty restricts local updates from diverging too far from the global model, stabilizing convergence across heterogeneous clinical devices and non-IID data distributions.

• Solves systems heterogeneity where clients have varying compute budgets
• Allows partial work from straggler devices
• Critical for cross-device healthcare deployments with mixed hardware

Uses control variates to correct for client drift during local training. Unlike FedAvg, SCAFFOLD tracks the difference between global and local update directions, significantly reducing the number of communication rounds required for convergence.

• Addresses the fundamental client-drift problem in FedAvg
• Maintains a server-side control variate and client-side variants
• Particularly effective when clinical data is highly non-IID across hospitals

A class of robust aggregation rules ensuring global model convergence even when a subset of nodes submits arbitrary, malicious, or corrupted updates. Essential for healthcare networks where compromised devices or adversarial actors may exist.

• Krum: Selects the update with minimal sum of squared distances to neighbors
• Trimmed Mean: Discards extreme values per coordinate before averaging
• Median Aggregation: Uses coordinate-wise median instead of mean

A cryptographic protocol enabling the server to compute the sum of encrypted client updates without inspecting any individual contribution. Ensures that even the aggregator cannot access raw model updates, providing strong data privacy guarantees.

• Uses secret sharing and pairwise masking
• Server only sees the aggregated result, never individual gradients
• Foundational for HIPAA-compliant multi-institutional training

Injects calibrated statistical noise into the model update process, providing formal mathematical guarantees that individual patient data cannot be inferred from the global model. Balances the privacy-utility tradeoff through the epsilon parameter.

• Gaussian or Laplacian noise addition to clipped gradients
• Provides (ε, δ)-differential privacy guarantees
• Essential for GDPR and HIPAA compliance in clinical federated learning

Decouples local client updates from global server optimization, allowing adaptive optimizers like Adam, Yogi, or SGD with momentum on the server side. Treats client updates as pseudo-gradients in a broader optimization framework.

• FedAdam: Adam optimizer applied server-side
• FedYogi: Adaptive learning rates for non-stationary settings
• Improves convergence speed over vanilla FedAvg in sparse participation regimes