Non-IID Data

Clients possess different proportions of target classes. In a mobile keyboard prediction scenario, one user might type about sports 80% of the time while another types about cooking 80% of the time. This label skew causes local models to overfit to client-specific priors, pulling the global model in conflicting directions during Federated Averaging (FedAvg). The global model often fails to generalize to minority classes on specific clients.

The marginal distribution of input features P(x) varies across clients, even if the conditional label distribution P(y|x) is shared. In handwriting recognition, different users have distinct stroke widths, slant angles, and character shapes for the same letter. This covariate shift forces the model to learn a highly variable input manifold, making it difficult to find a single set of weights that performs optimally for all local data distributions.

The relationship between features and labels P(y|x) differs across clients. In a global image classifier, a 'house' might be a brick building in one region and a bamboo hut in another. This concept drift is the most challenging form of non-IID data because a single global model cannot simultaneously map divergent inputs to the same label. It often requires personalized federated learning strategies like local fine-tuning or multi-task learning.

Clients contribute vastly different amounts of training data. A power user might generate 10,000 samples while a sporadic user generates only 50. In standard FedAvg, clients with more data exert disproportionate influence on the global model. Without careful weighting or client selection strategies, the global model overfits to heavy contributors and ignores the statistical patterns of low-data clients, reducing overall fairness and generalization.

The local data distribution on a single client changes over time. A user's typing patterns shift from work-related vocabulary during the day to personal messaging at night. This temporal heterogeneity means that even a perfectly converged local model becomes stale. Federated systems must incorporate continuous learning mechanisms and decay factors to prioritize recent data without suffering catastrophic forgetting of older, still-relevant patterns.

An extreme case where each client holds data from only a single class or a tiny subset of classes. In a federated digit classifier, one client might only have images of the digit '3' while another only has '7'. This pathological skew prevents any single local model from learning a decision boundary between classes. The global model must rely entirely on aggregation to synthesize a complete classifier, making it highly vulnerable to Byzantine failures and slow to converge.

The core problem underlying Non-IID data, where local client datasets have different distributions. This manifests as label distribution skew (some clients have mostly class A, others class B), feature distribution skew (different handwriting styles for the same digit), or concept drift (the same label means different things to different clients). Statistical heterogeneity causes local minima to diverge from the global optimum, slowing convergence.

The foundational algorithm that combines locally trained model weights on a central server by computing a weighted average. While simple, FedAvg struggles significantly with Non-IID data. When client updates point in conflicting directions due to heterogeneous distributions, naive averaging can lead to weight divergence and catastrophic forgetting of the global objective.

A paradigm shift that accepts statistical heterogeneity rather than fighting it. Instead of forcing a single global model, it creates tailored local models for individual clients. Techniques include:

• Model interpolation: Mixing global and local model weights
• Meta-learning: Finding an initial model that adapts quickly to local data
• Multi-task learning: Treating each client as a separate but related task

The scheduling mechanism that determines which subset of available devices participates in a given training round. In Non-IID settings, random selection can exacerbate bias. Advanced strategies include importance sampling to select clients whose data distributions better represent the population, or clustering-based selection that groups clients by distribution similarity before aggregation.

A proximal optimization framework designed to handle systems and statistical heterogeneity. FedProx adds a proximal term to the local objective function that penalizes large deviations from the global model. This stabilizes training when clients have highly Non-IID data or varying computational capabilities, allowing partial updates from stragglers instead of dropping them entirely.

The phenomenon where the statistical relationship between input features and target labels changes over time. In federated settings, this creates temporal Non-IID challenges where a client's data distribution shifts between training rounds. Detection requires monitoring prediction confidence and data distribution metrics at the edge, triggering model adaptation or retraining when drift exceeds thresholds.