Non-IID Data

Occurs when the marginal probability distribution of labels P(y) varies across clients. A base station in a stadium will see vastly different traffic categories than one in a residential area.

• Example: Client A has 90% video streaming labels, Client B has 90% IoT telemetry labels
• Impact: Local models overfit to dominant classes, causing the global model to diverge
• Mitigation: FedProx with proximal regularization or class-balanced client sampling

The marginal distribution of input features P(x) differs across clients, even if the conditional label distribution P(y|x) remains consistent. This is the covariate shift problem in distributed settings.

• Example: Urban base stations observe high signal-to-noise ratios, while rural stations see predominantly weak signals
• Impact: A model trained on urban features fails to generalize to rural signal conditions
• Mitigation: Domain adaptation layers or feature normalization per client

The conditional relationship P(y|x) itself varies between clients, meaning the same input feature maps to different labels in different locations. This is the most severe form of non-IID data.

• Example: A specific interference pattern indicates congestion in one cell but a hardware fault in another
• Impact: Global model averaging becomes mathematically invalid as local optima point in conflicting directions
• Mitigation: Multi-task learning or personalized federated learning with client-specific model heads

The volume of training data varies by orders of magnitude across clients, creating an implicit weighting problem during aggregation where data-rich clients dominate the global model.

• Example: A macro cell processes 10TB of daily traffic data while a small cell generates only 50MB
• Impact: The global model overfits to high-volume clients, erasing patterns learned from smaller ones
• Mitigation: Weighted federated averaging with capped client contributions or data augmentation for low-volume clients

The statistical properties of local datasets evolve over time at different rates across clients, creating a moving target for the global model. This is distinct from static non-IID data and compounds convergence instability.

• Example: A base station near a highway experiences rush-hour traffic patterns that shift seasonally
• Impact: Stale local updates poison the global model when aggregation frequency is too low
• Mitigation: Sliding window training, decay-weighted aggregation, or continual learning with elastic weight consolidation

Quantifying the degree of non-IIDness between client distributions is essential for diagnosing convergence problems. Common metrics include the earth mover's distance and Jensen-Shannon divergence on label histograms.

• Earth Mover's Distance (Wasserstein): Measures the minimum cost to transform one distribution into another
• Jensen-Shannon Divergence: A symmetric, smoothed version of KL divergence bounded between 0 and 1
• Practical Use: Clustering clients by distribution similarity before training to create more homogeneous sub-federations

The formal condition where local client data distributions are not uniform. In Non-IID scenarios, the probability distributions P_i(x, y) vary significantly across clients, violating the IID assumption of standard SGD. This manifests as:

• Label distribution skew: Some clients have mostly class A, others class B.
• Feature distribution skew: The same label looks different across clients (e.g., different handwriting styles).
• Quantity skew: Clients hold vastly different amounts of data.

The strategy for choosing which clients participate in a training round. Random selection can exacerbate Non-IID issues by over-representing skewed distributions. Advanced strategies include:

• Importance sampling: Selecting clients whose data distributions are most beneficial for current model improvement.
• Clustered sampling: Grouping clients by data similarity to ensure diverse representation in each round.

Quantifying the difference between local and global data distributions is critical for diagnosis. Common metrics include:

• Earth Mover's Distance (EMD): Measures the cost of transforming one distribution into another, often used to quantify label distribution skew.
• KL Divergence: Measures information loss when approximating the global distribution with a local one. High divergence directly correlates with convergence instability and degraded global model accuracy.

A core tension in Non-IID settings. A purely global model may perform poorly on outlier clients. Solutions include:

• Multi-task learning: Treating each client as a separate task.
• Model interpolation: Mixing a global model with a locally fine-tuned model.
• Clustered federated learning: Training separate models for distinct client clusters. The goal is to balance a generalized global model with personalized local performance.

When a global model trained on Non-IID data sequentially overrides knowledge from previous client updates. If client distributions are highly skewed, the model may 'forget' how to perform well on data from earlier clients after aggregating updates from later ones. Mitigation strategies include rehearsal buffers (storing proxy data) or elastic weight consolidation to protect parameters important for previous tasks.