SCAFFOLD

The core innovation of SCAFFOLD is the maintenance of control variates (c_i for each client, c for the server). These variates estimate the update direction of the global model versus the local model. During local training, the gradient is corrected by (c - c_i), effectively pulling the local update back toward the global optimum. This prevents the local model from drifting too far toward its local minimum, which is the fundamental flaw of standard FedAvg on non-IID data.

Client drift occurs when local models diverge from the global optimum after multiple local SGD steps on heterogeneous data distributions. SCAFFOLD eliminates this drift by incorporating the gradient correction term directly into the local loss function. The result is that even with many local epochs, the local updates remain aligned with the global objective. This is particularly critical in healthcare settings where clinical data distributions vary wildly across institutions due to different patient demographics, equipment, and diagnostic protocols.

By correcting for drift, SCAFFOLD allows clients to perform many more local steps without diverging. Empirical results show SCAFFOLD can reduce the required communication rounds by 2x to 5x compared to FedAvg on non-IID data partitions. This is achieved without increasing per-round communication cost significantly—the control variate is the same size as the model itself, transmitted once per round alongside the model update. For bandwidth-constrained hospital networks, this efficiency is transformative.

SCAFFOLD functions as a variance-reduced stochastic gradient method in the federated setting. The control variates serve as a bias correction that reduces the variance of local stochastic gradients. This connects SCAFFOLD theoretically to SVRG and SAGA in classical optimization. The reduced variance means local updates are more consistently pointed toward the true global minimum, enabling the use of higher learning rates and faster convergence without the instability that plagues FedAvg in heterogeneous environments.

SCAFFOLD requires both the server and each client to maintain persistent state across rounds:

• Server state (c): The running average of all client control variates, updated each round as c ← c + (1/N) * Σ Δc_i
• Client state (c_i): A local control variate updated after each round of local training, representing the local gradient direction This dual-state architecture adds a minor storage overhead (one model-sized vector per client) but enables the algorithm's superior convergence properties. The state is not privacy-sensitive as it captures gradient directions, not raw data.

SCAFFOLD is explicitly designed for statistically heterogeneous (non-IID) data partitions, which are the norm in healthcare federated learning. Unlike FedAvg, which can diverge or stall when client data distributions differ significantly, SCAFFOLD provably converges at a rate independent of data heterogeneity. This makes it the preferred choice for cross-silo medical applications where each hospital's patient cohort represents a distinct distribution of diseases, demographics, and imaging protocols.