Non-IID Data

A fundamental violation of identical distribution where the marginal probability of labels P(y) varies dramatically across clients. One hospital may predominantly treat diabetic retinopathy cases while another specializes in glaucoma, creating concept drift across the network.

• Example: Client A has 90% positive cases, Client B has 5% positive cases
• Impact: Local models overfit to their dominant class, causing the global model to oscillate rather than converge
• Mitigation: FedProx adds a proximal term to constrain local updates; FedAvg with weighted aggregation partially compensates for class imbalance

Occurs when the marginal distribution of input features P(x) differs across clients, even if the conditional relationship P(y|x) remains consistent. In medical imaging, different hospitals use scanners from different manufacturers with varying resolution, contrast, and noise profiles.

• Example: Client A uses 3T MRI scanners; Client B uses 1.5T scanners with different pulse sequences
• Impact: The global model learns spurious correlations tied to site-specific acquisition artifacts rather than true pathology
• Mitigation: Federated domain adaptation and feature alignment techniques normalize latent representations before aggregation

The conditional distribution P(x|y) varies across clients, meaning the same label manifests through different feature patterns. A 'positive for pneumonia' label may correspond to bacterial consolidation patterns in one hospital and viral interstitial patterns in another due to demographic and epidemiological differences.

• Example: Stroke imaging at a comprehensive stroke center vs. a rural emergency department with different patient demographics
• Impact: The global model fails to learn a unified decision boundary, producing high variance in predictions across sites
• Mitigation: Personalized federated learning (pFL) allows local model heads to specialize while sharing feature extractor layers

A violation of the independent sampling assumption where clients contribute vastly different numbers of training samples. A large academic medical center may hold millions of records while a community clinic contributes only hundreds, creating statistical power asymmetries.

• Example: Client A contributes 500,000 samples; Client B contributes 500 samples
• Impact: Naive averaging drowns out the signal from smaller clients, potentially erasing rare disease patterns only present in specialized clinics
• Mitigation: Weighted aggregation proportional to local dataset size, combined with FedProx to prevent large clients from dominating the update direction

A dynamic form of non-IID data where the underlying data-generating process evolves over time, causing dataset staleness. A model trained on pre-pandemic chest X-rays will encounter fundamentally different feature distributions when deployed during a novel viral outbreak.

• Example: Seasonal variation in influenza presentation; sudden shifts during a pandemic
• Impact: Static federated models silently degrade in production as local distributions drift away from the training distribution
• Mitigation: Continuous federated learning with drift detection triggers model retraining; federated online learning updates models incrementally as new data arrives

A particularly severe form of feature skew where entire data modalities present in the global distribution are absent from specific clients. One hospital may collect genomic data alongside imaging, while another only has radiology reports, creating structural heterogeneity in the feature space.

• Example: Client A has imaging + genomics + EHR; Client B has imaging only
• Impact: Standard horizontal federated learning fails because the feature spaces are not aligned
• Mitigation: Vertical federated learning with entity alignment, or federated transfer learning to map heterogeneous feature spaces into a shared latent representation

The distributed systems property that ensures correct consensus despite arbitrary node failures or malicious behavior. In federated learning, BFT aggregation rules replace simple averaging to tolerate clients that send corrupted updates.

• Krum: Selects the single update most similar to its neighbors
• Median/Trimmed Mean: Coordinate-wise robust statistics
• BFT is essential when Non-IID data makes it difficult to distinguish a malicious gradient from a genuinely divergent local optimum
• Derived from the Lamport-Shostak-Pease Byzantine Generals Problem