Federated Averaging (FedAvg)

The core mechanism of FedAvg is the execution of multiple local SGD steps on each client before communication occurs. Unlike traditional distributed training where gradients are exchanged after every batch, FedAvg allows clients to compute model deltas over several epochs on their local data. This significantly reduces the communication-to-computation ratio, as the number of synchronization rounds is decoupled from the number of local optimization steps. The local update rule follows: w_{t+1} = w_t - η∇F_k(w_t), where F_k is the local objective on client k.

The server aggregates client updates using a weighted average proportional to the size of each client's local dataset. The global model update is computed as: w_{t+1} = Σ (n_k / n) * w^k_{t+1}, where n_k is the number of samples on client k and n is the total samples across all selected clients. This weighting scheme ensures that clients with more data exert proportionally greater influence on the global model, providing a statistically principled aggregation mechanism that converges to the centralized empirical risk minimizer under IID assumptions.

When local datasets are non-IID (heterogeneous), each client's local model drifts toward its own local optimum during multiple SGD steps. This client drift causes the averaged global model to diverge from the true global optimum, potentially slowing convergence or degrading final accuracy. The drift magnitude is proportional to the number of local epochs E and the degree of statistical heterogeneity. This is the primary challenge FedAvg faces in real-world deployments, motivating variants like FedProx and SCAFFOLD that add regularization or control variates to correct the drift.

FedAvg operates with a client fraction C (0 < C ≤ 1), selecting only a random subset of available clients per round. This partial participation is essential for scalability in cross-device settings with millions of clients, where waiting for all devices would be infeasible. The stochastic nature of client selection introduces additional variance into the aggregation, but also provides a regularization effect that can improve generalization. Typical values for C range from 0.1 to 0.5 in production deployments.

Each FedAvg round follows a strict synchronous protocol:

• Step 1: Server broadcasts current global model w_t to selected clients
• Step 2: Clients initialize local model with w_t and perform E local epochs of SGD
• Step 3: Clients transmit their updated model weights w^k_{t+1} back to the server
• Step 4: Server computes the weighted average and updates the global model This synchronous barrier ensures consistent aggregation but makes the system vulnerable to stragglers—slow clients that delay the entire round.

FedAvg's performance is highly sensitive to three key hyperparameters:

• Local epochs E: More epochs reduce communication but increase client drift. Values typically range from 1 to 20.
• Local batch size B: Affects gradient noise and convergence speed. Larger batches reduce variance but increase computation.
• Client fraction C: Controls the trade-off between round latency and update variance. Tuning these parameters requires balancing communication efficiency against convergence stability, with optimal values depending heavily on the degree of data heterogeneity across clients.

The primary deployment topology for FedAvg in healthcare, where a small number of reliable institutional clients (e.g., hospitals) participate in all training rounds. Unlike cross-device settings with millions of unreliable smartphones, cross-silo assumes stateful clients with significant local compute and curated datasets. This topology allows FedAvg to run synchronous rounds without severe straggler issues, making it the default architecture for multi-institutional diagnostic model training.

The primary challenge to FedAvg convergence in real-world medical settings. Patient populations vary dramatically across hospitals—a rural clinic sees different disease prevalence than an urban research center. This statistical heterogeneity causes client drift, where local SGD steps pull models toward divergent local optima. FedAvg's weighted averaging partially mitigates this, but severe label distribution skew or concept shift requires algorithmic modifications like FedProx or SCAFFOLD to restore stability.

A cryptographic layer that pairs with FedAvg's weighted averaging step to prevent the central server from inspecting individual client model updates. Using secure multi-party computation or homomorphic encryption, the server computes the weighted average of encrypted model deltas without ever seeing a single hospital's contribution in plaintext. This transforms FedAvg from a privacy-conscious protocol into a mathematically privacy-guaranteed one, essential for HIPAA and GDPR compliance in cross-border medical research.

The scheduling logic that determines which hospitals participate in each FedAvg communication round. Random selection is the baseline, but healthcare deployments often use bandwidth-aware scheduling to prioritize clients with available network capacity or data-quality-weighted selection to ensure representative clinical distributions. Poor client selection can introduce sampling bias that skews the global model toward over-represented patient demographics.

A mathematical framework applied to FedAvg's weight updates before transmission. By clipping per-client model deltas and injecting calibrated Gaussian noise into the aggregated result, the central server can bound the information leakage about any single patient's record. This converts FedAvg's implicit privacy (data stays local) into explicit, provable privacy with a measurable epsilon budget, satisfying institutional review boards and regulatory auditors.

The distributed auditing process that validates a FedAvg-trained global model without centralizing test data. Each hospital evaluates the model on its local holdout set and reports only aggregate metrics back to the server. This preserves patient data locality while enabling bias detection across demographic subgroups. Without federated evaluation, a model that appears accurate globally may silently fail on underrepresented populations at specific clinical sites.