Label Distribution Skew

The most extreme form of label skew occurs when specialized clinical sites possess data for only a subset of classes. A rare disease center may have thousands of positive cases but zero examples of common conditions, while a general practice clinic shows the inverse distribution.

• Complete class absence: Some clients lack entire label categories
• Long-tail distributions: A few classes dominate while others appear rarely
• Mutually exclusive label sets: Clients A and B may have no overlapping classes

This violates the standard IID assumption that every client's label distribution approximates the global prior.

A related but distinct phenomenon where clients hold drastically different volumes of labeled data per class. A major academic medical center may contribute 100,000 labeled chest X-rays while a rural clinic contributes only 200.

• Sample size disparity: Orders-of-magnitude differences in per-class counts
• Dominant client bias: The global model overfits to data-rich institutions
• Statistical noise amplification: Small clients introduce high-variance gradient updates

Quantity skew often co-occurs with distributional skew, compounding the optimization challenge.

When local label distributions diverge, the FedAvg aggregation algorithm produces a global model that fails to converge to the true optimum. Each client's local SGD steps pull the model toward its own biased local minimum.

• Client drift: Local updates diverge from each other during training
• Weight divergence: The Euclidean distance between client models grows with each communication round
• Accuracy degradation: Global test accuracy can drop by 20-40% compared to IID baselines

The mathematical root cause is that the local empirical risk minimizer differs substantially from the global empirical risk minimizer under heterogeneous P(y).

Instead of sharing model weights, clients exchange compact class-representative vectors called prototypes. Each prototype is the mean embedding of all samples belonging to a class in the client's feature space.

• Communication efficiency: Prototypes are orders of magnitude smaller than full model updates
• Natural skew handling: Prototypes normalize for class frequency differences
• Heterogeneous architecture support: Clients can use different model architectures

The global server aggregates prototypes per class, creating a decision boundary that is robust to label imbalance across sites.

In healthcare federated learning, label skew is not a bug but a structural reality. A cardiology department's EHR data is naturally enriched for cardiac ICD-10 codes, while an oncology center's labels cluster around neoplasm classifications.

• Referral patterns: Tertiary care centers receive filtered patient populations
• Screening program effects: Mammography centers have elevated breast cancer label rates
• Geographic disease prevalence: Endemic disease labels concentrate regionally

Ignoring this skew produces models that underperform on rare diseases at general hospitals and over-diagnose common conditions at specialty centers.

The fundamental challenge in federated learning where local client datasets are not independent and identically distributed (non-IID). This variation encompasses differences in feature distributions, label distributions, and the relationship between them. In healthcare, this is the norm, not the exception, as patient populations, disease prevalence, and clinical practices vary widely across institutions.

A type of non-IID data where the marginal distribution of input features P(x) differs across clients, even if P(y|x) is shared. Common causes in healthcare include:

• Different patient demographics (age, ethnicity, socioeconomic status)
• Varied medical device manufacturers (Siemens vs. GE MRI scanners)
• Distinct acquisition protocols (slice thickness, contrast agents) This skew can cause a global model to learn spurious correlations tied to a site's specific imaging pipeline rather than true pathology.

A communication-efficient method specifically designed to handle label distribution skew. Instead of sharing full model weights, clients exchange compact class-representative vectors (prototypes) computed from their local data. The global server aggregates these prototypes to form a robust decision boundary. This approach naturally handles scenarios where one hospital specializes in rare diseases and another sees only common ailments, as the prototype for each class is built from all sites that have examples of it.

A strategy that partitions clients into groups with similar data distributions and trains a separate model for each cluster. This mitigates the negative impact of extreme statistical heterogeneity by avoiding a one-size-fits-all global model. For example, a network of oncology centers and a network of general hospitals would form distinct clusters, each learning a specialized model that reflects their shared label distribution, rather than forcing a compromise that degrades performance for both.

A framework that trains personalized models for each client simultaneously. It learns shared representations across all clients while allowing for client-specific model parameters to handle local data biases. This is particularly effective for label distribution skew because the shared layers capture universal diagnostic features, while the personalized heads adapt to each hospital's unique class priors, preventing a rare-disease classifier from being overwhelmed by a site with no positive examples.

An optimization framework that learns data representations which elicit the same optimal classifier across all training clients. The goal is to discover causal relationships that are robust to spurious correlations introduced by label distribution skew. For instance, if a model learns to associate a specific hospital's imaging artifact with a disease because that hospital has a high prevalence, FedIRM penalizes such environment-specific predictors, forcing the model to rely on true pathological features.