Federated Averaging

The core operation of FedAvg is a weighted arithmetic mean of local model parameters. Each client's contribution is scaled proportionally to the size of its local dataset (n_k / n_total). This ensures that clients with more representative data exert a proportionally greater influence on the global model, while smaller datasets still contribute valuable signal without being drowned out. The update rule is: w_global = Σ (n_k / n) * w_k, where w_k represents the model weights trained locally on client k.

FedAvg generalizes Stochastic Gradient Descent (SGD) to a distributed setting. Instead of transmitting gradients after every batch, each client performs multiple local epochs of SGD on its private data. Only the final model weights—not raw gradients or data—are sent to the server. This reduces communication rounds by orders of magnitude compared to naive distributed SGD. The hyperparameter E (local epochs) controls the compute-to-communication trade-off:

• E=1: Single local epoch, closer to standard distributed SGD
• E=∞: Full local convergence, extreme communication savings but risk of client drift

When local datasets are non-IID (not independently and identically distributed), each client's local SGD trajectory diverges toward a different local optimum. This phenomenon, called client drift, causes the averaged global model to converge to a suboptimal point far from the true centralized optimum. FedAvg's convergence guarantees degrade significantly under heterogeneous data distributions—a critical concern in genomic consortia where each hospital may have distinct patient demographics and sequencing protocols.

FedAvg is designed to minimize uplink bandwidth—the bottleneck in federated systems. By transmitting only model weights after multiple local epochs, it achieves a compression ratio proportional to the number of local steps. Additional optimizations include:

• Partial participation: Only a random fraction C of clients train in each round
• Gradient sparsification: Transmitting only the top-k magnitude weight updates
• Quantization: Reducing weight precision from 32-bit floats to 8-bit integers These techniques make FedAvg viable for cross-silo genomic federations with limited inter-institutional bandwidth.

FedAvg provably converges to the global optimum under convex loss functions with IID data. For non-convex objectives like deep neural networks, convergence is empirically robust but theoretically more fragile. Key factors affecting convergence:

• Learning rate decay: Critical for stabilizing late-stage training
• Local epoch count: Too many local epochs on non-IID data causes divergence
• Client sampling strategy: Uniform random sampling can bias the model toward over-represented clients
• Server momentum: Applying momentum to the global weight update (FedAvgM) significantly accelerates convergence in practice

FedAvg provides a baseline layer of data locality privacy—raw genomic sequences never leave their institution of origin. However, model weight updates can still leak information through gradient inversion attacks. To strengthen guarantees, FedAvg is commonly combined with:

• Differential Privacy: Adding calibrated Gaussian noise to weight updates before transmission
• Secure Aggregation: Using multi-party computation to ensure the server only sees the aggregated sum, never individual updates This layered defense is essential for HIPAA-compliant and GDPR-compliant genomic federations.

The overarching decentralized machine learning paradigm where a shared global model is trained across multiple clients holding local data samples, without exchanging the raw data itself. Federated Averaging is the most common optimization algorithm used to coordinate this process.

• Goal: Train a model without centralizing sensitive data.
• Process: A server sends a model to clients, they train locally, and only model updates are returned.
• Contrast: The opposite of traditional data-center training.

A privacy-preserving cryptographic protocol that allows a central server to compute the sum of model updates from multiple clients without inspecting any individual client's contribution. This is a critical companion to Federated Averaging.

• Mechanism: Uses secret sharing to mask individual updates.
• Benefit: The server only sees the final averaged result, not any single hospital's gradient.
• Threat Model: Protects against an honest-but-curious server.

A mathematical framework providing a provable guarantee of privacy by injecting calibrated statistical noise into query results or model updates. In Federated Averaging, noise is often added to the aggregated update before applying it to the global model.

• Key Parameter: Epsilon (ε) quantifies the privacy loss budget.
• Technique: Gaussian noise is added to the averaged gradient.
• Trade-off: Higher noise increases privacy but degrades model accuracy.

A federated topology involving a small, reliable number of institutional participants, such as hospitals or genomic research centers. This is the primary setting for Federated Averaging in healthcare.

• Participants: Typically 2–100 organizations.
• Characteristics: Reliable connectivity, substantial local compute, and large, curated datasets.
• Contrast: Unlike cross-device FL, clients are stateful and identified.

A data distribution characteristic where local datasets on different clients are not independently and identically distributed. This is a fundamental challenge for Federated Averaging in genomics, where patient demographics and sequencing protocols vary wildly between hospitals.

• Impact: Can cause the global model to diverge or converge slowly.
• Genomic Example: One hospital may sequence rare diseases, another focuses on oncology.
• Mitigation: Proximal terms and personalized FL variants are used to combat this drift.

A critical optimization objective focused on minimizing the size and frequency of data transmitted between clients and the server. Genomic models like DNABERT have millions of parameters, making naive transmission of full weights a bandwidth bottleneck.

• Techniques: Gradient compression, sparsification, and quantization.
• Federated Averaging Role: The algorithm reduces communication by performing multiple local epochs before sending an update.
• Trade-off: More local computation reduces communication rounds but risks local overfitting.