Federated Non-IID

Occurs when different hospitals treat vastly different patient populations, leading to a non-uniform distribution of labels across clients. For example, a specialized cardiac hospital may have a high prevalence of heart disease labels, while a general clinic has mostly healthy patients. This causes local models to overfit to their dominant local classes, pulling the global model in conflicting directions during aggregation. FedProx and SCAFFOLD are algorithms specifically designed to correct for this drift by adding proximal terms or control variates.

Also known as covariate shift, this occurs when the same diagnostic label looks different across sites. An MRI of pneumonia taken on a 1.5T scanner in a rural clinic has different texture and resolution characteristics than one from a 3T scanner in an academic medical center. The model learns spurious correlations between scanner hardware and pathology. Techniques like domain adversarial training and feature alignment are used to force the model to learn scanner-invariant representations.

A more severe form of skew where the statistical relationship between features and labels differs across clients. For instance, a specific genetic biomarker may be highly predictive of a drug response in one ethnic population but irrelevant in another. A global model that averages these conflicting signals will fail for both groups. Personalized federated learning via multi-task learning or local fine-tuning is often the only viable solution to preserve per-client accuracy.

A major academic medical center may contribute millions of records, while a community hospital contributes only a few thousand. Naive FedAvg weighting by local dataset size can cause the global model to be dominated by the largest client, erasing rare but clinically vital patterns from smaller sites. FedNova and other normalized averaging techniques correct for this by accounting for the varying number of local SGD steps, not just raw data volume.

Clinical data distributions are not static; they evolve over time due to new treatment protocols, seasonal disease patterns, or updated diagnostic criteria. A model trained on pre-pandemic chest X-rays will catastrophically fail on COVID-19 pneumonia cases. In a federated setting, different clients may experience these shifts at different times. Federated continual learning strategies, including rehearsal buffers and elastic weight consolidation, are required to prevent catastrophic forgetting of historical patterns.

Measured by the Earth Mover's Distance (EMD) between client data distributions. High EMD between clients indicates that their local optima are geometrically far apart in the loss landscape. Averaging these distant points creates a global model that sits in a high-loss region, performing worse than any individual local model. Hierarchical aggregation clusters clients with similar distributions before global merging, reducing the variance of the aggregated update.

The fundamental condition where local client data distributions differ significantly from the global population. In healthcare, this manifests as label distribution skew (Hospital A has mostly diabetic patients, Hospital B has mostly cardiac patients), feature distribution skew (different imaging equipment vendors), and concept drift (varying diagnostic criteria across regions). This violates the standard IID assumption of centralized machine learning.

The pathological outcome of training on non-IID data without mitigation. Local models drift toward their own local optima rather than converging to a shared global optimum. Key indicators include:

• Weight divergence: Local model parameters move in conflicting directions
• Performance degradation: Global model accuracy drops by 20-40% compared to IID baselines
• Client drift: The phenomenon where local updates point away from the global objective

A paradigm shift that embraces non-IID data rather than fighting it. Instead of forcing a single global model, pFL learns client-specific models that share common knowledge while adapting to local distributions. Key approaches include:

• Multi-task learning: Treat each client as a separate task with shared representations
• Model interpolation: Blend global and local models via learned mixing weights
• Federated meta-learning: Train a model that can quickly adapt to new client distributions

Privacy-preserving techniques to artificially balance non-IID distributions across clients. Methods include federated GANs that generate synthetic samples to fill distribution gaps, mixup augmentation that creates virtual training examples by interpolating between client data points, and data sharing where a small curated subset of globally balanced data is distributed to all clients. These approaches reduce statistical skew without exposing raw patient records.

Monitoring techniques to detect and quantify the impact of non-IID data during training. Key metrics include:

• Gradient variance: High variance across client updates signals distribution mismatch
• Local-global weight divergence: Euclidean distance between local and global model parameters
• Earth Mover's Distance: Quantifies the statistical distance between client data distributions
• Client contribution entropy: Measures how uniformly clients influence the global model