Non-IID Data

Also known as label skew or prior probability shift, this occurs when the marginal distribution of labels P(y) varies significantly across clients. In genomic federated learning, one hospital may have a high prevalence of a specific rare disease cohort, while another has none. This causes the local stochastic gradient to become a biased estimator of the global gradient, leading to client drift and severe divergence in Federated Averaging (FedAvg). Mitigation strategies include FedProx, which adds a proximal term to the local objective, and scaffold, which corrects for client drift using control variates.

Also known as covariate shift, this occurs when the marginal distribution of features P(x) differs across clients. In genomic contexts, this manifests as different sequencing platforms (Illumina vs. PacBio), varying read depths, or distinct population ancestry structures in local biobanks. A model trained on one sequencing technology may fail to generalize to another. Techniques like domain adversarial training and feature alignment via maximum mean discrepancy (MMD) are used to learn invariant representations that ignore platform-specific artifacts.

Also known as conditional distribution skew, this occurs when the relationship between features and labels P(y|x) differs across clients. In genomics, the same genetic variant may have different phenotypic penetrance in different populations due to gene-environment interactions or epistatic effects. This violates the core IID assumption that a single global model can serve all clients. Personalized federated learning approaches, such as multi-task learning or model interpolation via APFL, explicitly learn a mixture of global and local parameters to handle this divergence.

Also known as unbalancedness, this refers to the significant variance in the volume of local data across clients. A large academic medical center may hold millions of patient genomes, while a rural clinic contributes only a few hundred. In standard FedAvg, weighting updates by dataset size can cause the global model to be dominated by data-rich clients, erasing rare signals from smaller cohorts. Techniques like FedNova normalize local updates to eliminate objective inconsistency, and q-FedAvg explicitly optimizes for fair resource allocation across clients.

In cross-device federated learning, the local data distribution on a client changes over time. For genomic surveillance systems tracking pathogen evolution, the distribution of viral sequences on a local device shifts as new variants emerge. This violates the IID assumption across time steps. Continual federated learning and rehearsal-based methods that store a coreset of representative historical samples are required to prevent catastrophic forgetting of older variants while adapting to new ones.

Quantifying the degree of non-IIDness is essential for diagnosing convergence issues. The Earth Mover's Distance (EMD) between client label distributions provides a scalar measure of label skew. For feature skew, centered kernel alignment (CKA) or Fréchet distance between client feature representations can identify clusters of compatible clients. These metrics inform client selection strategies, where a coordinator preferentially samples clients whose data distributions are mutually beneficial for the current training phase, avoiding harmful gradient interference.