Elastic Weight Consolidation (EWC)

The fundamental problem that Elastic Weight Consolidation is designed to solve. When a neural network is trained sequentially on new tasks or patient data distributions, its performance on previously learned tasks can abruptly and completely degrade. This occurs because gradient updates for the new task overwrite the weights critical for the old task. In medical devices, this could mean a model that learns to detect a new arrhythmia suddenly loses its ability to identify atrial fibrillation. EWC mitigates this by identifying and protecting the parameters most important to prior tasks.

A regularization-based approach similar to EWC that addresses catastrophic forgetting. While EWC uses the Fisher Information Matrix to estimate parameter importance after training, Synaptic Intelligence computes importance along the entire learning trajectory in an online fashion. Key differences:

• Accumulates importance during the training process itself
• Does not require a separate post-hoc computation phase
• Penalizes changes to parameters that contributed most to reductions in the loss function This method is particularly relevant for on-device scenarios where storing the Fisher matrix for every past task is memory-prohibitive.

A complementary strategy to EWC that explicitly stores and rehearses samples from previous tasks. Instead of only constraining weights, the model periodically retrains on a small episodic memory buffer of past examples. Variants include:

• Experience Replay: Storing raw data samples from prior tasks
• Generative Replay: Training a generative model to produce synthetic past examples, avoiding raw data storage In healthcare federated learning, generative replay is preferred because storing actual patient data on-device creates privacy risks. Combining EWC with a small replay buffer often yields superior retention compared to either method alone.

An architectural approach to continual learning that sidesteps forgetting entirely by freezing previously trained network columns and adding new lateral connections for each new task. Unlike EWC, which constrains a single model, Progressive Networks:

• Instantiate a new neural network column per task
• Connect new columns to all prior columns via learned adapter layers
• Never modify previously learned weights This guarantees zero forgetting but at the cost of linear growth in parameters with each new task. For resource-constrained medical edge devices, this parameter explosion makes EWC's single-model approach more practical.

An extension of standard EWC that removes the requirement for discrete task boundaries. Original EWC needs a clear signal when one task ends and another begins to compute the Fisher matrix. Online EWC adapts this for continuous, task-free learning streams by:

• Maintaining a running estimate of parameter importance
• Updating the Fisher information incrementally as new data arrives
• Applying a decaying weight to older Fisher estimates This variant is essential for medical wearables that continuously encounter evolving patient data patterns without explicit task delineation, such as a glucose monitor adapting to a patient's changing metabolic profile over months.

The intersection of federated learning and continual learning where EWC operates at both the local and global levels. In this paradigm:

• Each edge device uses EWC to retain personalized knowledge during local fine-tuning
• The central server applies EWC-like constraints when aggregating updates to prevent global forgetting
• Devices can learn new tasks collaboratively without a central dataset This dual-level protection is critical for multi-institutional healthcare networks where a global diagnostic model must continuously incorporate new disease classes from different hospitals while preserving accuracy on all previously learned conditions.