Federated Model Divergence

The fundamental mechanism of divergence where local models overfit to their non-IID data shards during local training epochs. Each client's stochastic gradient descent steps pull the model toward a local minimum that conflicts with the global objective.

• Occurs when local datasets have skewed label distributions (e.g., one hospital has 80% positive cases while another has 10%)
• Exacerbated by excessive local epochs before aggregation
• Measured by the gradient dissimilarity between local updates and the global update direction
• Manifests as weight divergence where model parameters cluster by client rather than converging to a shared optimum

The root cause of divergence: participating clients hold data drawn from fundamentally different probability distributions. In healthcare, this arises from varying patient demographics, equipment manufacturers, and clinical protocols.

• Covariate shift: Different hospitals image patients with different scanner vendors, producing varying pixel intensity distributions
• Label distribution skew: A specialized cardiac center sees far more positive cases than a general clinic
• Concept drift: The same diagnosis may manifest differently across populations (e.g., disease presentation varies by age cohort)
• Quantity skew: Some institutions contribute millions of records while others contribute thousands

Divergence produces oscillatory or chaotic global model behavior rather than smooth convergence. The aggregated model's loss function exhibits high variance across communication rounds.

• Global loss spikes occur when a client with extreme data distribution contributes an update
• The model may forget previously learned patterns as it adapts to the most recent client's data distribution
• Convergence curves show sawtooth patterns instead of monotonic improvement
• In severe cases, the global model performs worse than isolated local models trained without federation

A measurable signature of divergence where local stochastic gradients point in opposing directions across clients. When aggregated, these conflicting signals cancel out, stalling training.

• Quantified by negative cosine similarity between gradient vectors from different clients
• Arises when different clients' optimal decision boundaries are mutually exclusive
• Standard Federated Averaging (FedAvg) assumes gradients are aligned in expectation, an assumption violated under severe non-IID conditions
• Mitigation strategies include gradient clipping, variance reduction, and proximal regularization terms that penalize deviation from the global model

A severe form of divergence where the global model's internal feature representations degrade to capture only the dominant client's data patterns, losing the ability to generalize across the full population.

• The feature extractor layers converge to a narrow subspace that fails to encode minority class characteristics
• Particularly dangerous in healthcare where rare disease detection depends on preserving diverse representations
• Detected by monitoring inter-client feature divergence using centered kernel alignment or CKA similarity metrics
• Addressed through federated contrastive learning and representation matching objectives that explicitly preserve feature diversity

When federated training proceeds over sequential tasks or changing data distributions, divergence manifests as catastrophic forgetting—the global model abruptly loses performance on previously learned tasks.

• Occurs in federated continual learning scenarios where new hospitals join with novel disease categories
• The global model's weights are overwritten to accommodate new distributions at the expense of old ones
• Measured by backward transfer metrics that quantify performance degradation on earlier tasks
• Mitigated through elastic weight consolidation, memory replay buffers (using synthetic data), and dynamic architecture expansion that allocates new parameters for new tasks