Catastrophic Forgetting Mitigation

A regularization technique that identifies and slows down learning on weights critical to previous tasks. EWC computes the Fisher Information Matrix to estimate parameter importance, then adds a quadratic penalty that anchors important weights to their prior values during new task training.

• Prevents abrupt performance drops on old clinical sites
• Computationally efficient: adds only a regularization term to the loss
• Ideal for sequential local fine-tuning in federated settings

An online method that tracks each synapse's contribution to the overall loss reduction over a task's training trajectory. Unlike EWC, SI does not require computing second-order derivatives, making it more scalable for deep networks.

• Accumulates importance weights during training, not post-hoc
• Penalizes changes proportionally to each parameter's historical utility
• Well-suited for continual federated learning where tasks arrive sequentially

Stores a small representative subset of previous task data—called exemplars—and interleaves them with new data during training. This directly reinforces old knowledge while learning new patterns.

• Episodic memory: retains raw samples (subject to privacy constraints)
• Generative replay: trains a generative model to synthesize past data
• In federated settings, requires careful handling of patient data retention policies

Freezes previously learned model columns and adds lateral connections to new columns trained on new tasks. Each task gets its own dedicated parameters, completely eliminating forgetting at the cost of linear parameter growth.

• Zero interference between tasks
• Lateral connections enable positive transfer from prior knowledge
• Practical for small numbers of sequential clinical tasks

Uses the previous model as a teacher to generate soft targets for the current model during new task training. The student model is trained to match both the new task labels and the teacher's output distribution on old tasks.

• Preserves decision boundaries learned from prior data distributions
• Does not require storing raw historical data
• Commonly combined with Federated Model Distillation for decentralized settings

A distillation-based approach where the model's responses on new task samples are recorded before and after training. The original model's outputs serve as pseudo-labels for the old tasks, constraining the updated model to maintain prior performance.

• Only requires the current task dataset—no exemplar storage
• Jointly optimizes new task accuracy and old task stability
• Effective when old task data cannot be retained due to privacy regulations

A continual learning algorithm that identifies parameters critical to previously learned tasks and penalizes their modification during new training. In federated settings, EWC calculates the Fisher Information Matrix on the global model to determine weight importance. When a local client fine-tunes on site-specific data, the loss function includes a quadratic penalty that constrains changes to these high-importance weights.

• Prevents abrupt performance drops on the original global task distribution
• Does not require storing old patient data, preserving privacy
• Computationally efficient compared to rehearsal-based methods

An online variant of synaptic consolidation that tracks each parameter's contribution to the total loss reduction during a task. Unlike EWC, SI computes importance weights in an online fashion during training without requiring explicit Fisher information calculation. When adapting to a new local data distribution, SI applies a per-synapse regularizer proportional to the parameter's accumulated importance.

• Tracks a 'synaptic importance' pathway for every weight
• Reduces catastrophic forgetting in sequential federated rounds
• Well-suited for non-stationary clinical environments where task boundaries are blurred

A consolidation method that estimates parameter importance by measuring the sensitivity of the learned function's output to small weight perturbations. MAS computes importance using unlabeled local data, making it highly practical for federated healthcare settings where labels may be scarce. The importance weight for each parameter is proportional to the magnitude of the gradient of the squared L2-norm of the model output.

• Does not require labeled data for importance estimation
• Accumulates importance weights cumulatively across federated rounds
• Effective at preserving both classification and regression task performance

An architectural approach that instantiates a new neural network column for each new task while freezing previously learned columns. Lateral connections from prior columns to the new column enable transfer learning without modifying existing weights. In federated personalization, each client can add a local column that leverages the frozen global feature extractors.

• Completely eliminates catastrophic forgetting by design
• Introduces linear growth in parameters per new task
• Enables positive forward transfer from global to local representations

A privacy-compliant alternative to data rehearsal where the global model's logits on a public proxy dataset are stored and replayed during local fine-tuning. Instead of retaining raw patient data, clients distill the global model's predictive distribution and use it as soft targets to regularize local adaptation.

• Avoids storing any patient-level information
• Uses knowledge distillation as a form of functional regularization
• Requires a representative unlabeled public dataset for logit generation

A rehearsal-constrained optimization method that stores a small memory buffer of examples from previous tasks. During local training on new data, GEM projects the gradient such that the loss on buffered memories does not increase, enforcing a non-negative backward transfer constraint. In federated contexts, the memory buffer can consist of synthetic or proxy samples.

• Provides strong theoretical guarantees against forgetting
• Requires maintaining a memory buffer, which may be constrained on edge devices
• Effective when a small set of representative global samples can be legally retained