Federated Catastrophic Forgetting

The fundamental tension in federated continual learning. Plasticity is the model's ability to learn new clinical tasks from incoming institutional data streams. Stability is its ability to retain performance on previously learned tasks. Federated catastrophic forgetting occurs when plasticity dominates, causing the global model's weights to be overwritten by new, site-specific updates, erasing prior diagnostic capabilities. This is exacerbated by the sequential, non-IID nature of healthcare data silos.

A primary mechanism of forgetting in federated networks. When Institution A fine-tunes a foundation model on radiology reports and Institution B subsequently fine-tunes on pathology slides, the gradient updates from Institution B can destructively interfere with the weight configurations crucial for Institution A's task. This is not mere overfitting; it is a direct conflict in the loss landscape where optimizing for a new data distribution degrades performance on a prior one, even if the prior data is never seen again.

Unlike centralized continual learning, federated forgetting is driven by a temporal sequence of heterogeneous data silos. Each institution represents a distinct distribution shift. A model might first learn from a large urban hospital's diverse dataset, then be updated by a small rural clinic's data, causing it to forget rare urban pathologies. The order in which institutions contribute updates critically determines which knowledge is retained and which is erased, creating a path-dependent learning trajectory.

A measurable symptom of catastrophic forgetting in federated systems. As local models are trained on site-specific data, their weights drift from the global consensus. Without corrective mechanisms, this representation drift causes the feature extractors to specialize to the most recent tasks. A neuron that once activated for 'pulmonary embolism' in a chest X-ray model might be repurposed to detect 'atelectasis' after several rounds of local training, permanently altering the model's internal logic.

A key countermeasure that estimates the Fisher Information Matrix for each parameter to identify weights critical to previous tasks. During local training at a new institution, a quadratic penalty is applied to changes in these important weights, effectively 'freezing' the model's prior knowledge. In a federated context, this requires aggregating both the model weights and the importance matrices, allowing the global model to learn new clinical tasks while preserving performance on all previously encountered institutional datasets.

A strategy involving the storage and replay of representative samples from prior tasks. In privacy-preserving healthcare, this typically uses a federated generative replay approach: a generative model is collaboratively trained to produce synthetic, anonymized samples of past clinical data distributions. When a new institution joins, these synthetic samples are interleaved with the new local data during training, preventing the model's decision boundaries from collapsing onto only the new task's feature space.

A foundational paradigm where a model pre-trained on a large public corpus is distributed to institutions for fine-tuning on private clinical data. Only the task-specific final layers are aggregated centrally. This approach is highly susceptible to catastrophic forgetting on the original pre-training tasks when sequentially adapted to new, narrow clinical domains without proper stabilization techniques.

A 'learning to learn' approach designed explicitly to combat forgetting. The model is trained across diverse clinical tasks from multiple institutions to find an optimal initialization. This model-agnostic meta-learning (MAML) starting point allows rapid adaptation to a new task at a new hospital with very few data points, while naturally retaining prior knowledge due to its optimized parameter space.

A model compression technique that also serves as an anti-forgetting mechanism. A global 'teacher' model's knowledge is transferred to smaller 'student' models at each site by sharing only the teacher's output logits on a public or synthetic dataset. By replaying the teacher's soft labels, the student model is regularized against forgetting previously learned clinical tasks without accessing the original private data.

An adaptation of the classic EWC algorithm to a decentralized setting. During local training on a new task, a quadratic penalty is applied to changes in parameters deemed important for previous tasks. The Fisher Information Matrix is computed locally and aggregated centrally to identify these critical weights, effectively slowing learning on them to preserve prior clinical knowledge.

A technique that adds a penalty to the local training objective to prevent the feature representations learned at one institution from diverging too far from the global consensus. This ensures a semantically consistent embedding space across the network and acts as a soft barrier against representational drift, a primary cause of catastrophic forgetting in non-IID federated environments.

The statistical heterogeneity of decentralized clinical datasets is a primary accelerator of catastrophic forgetting. When local data distributions are not independent and identically distributed, sequential local updates can pull the global model in conflicting directions. Strategies like FedProx and SCAFFOLD correct for this client drift, indirectly stabilizing the global model against forgetting by ensuring more consistent local optimization.