Elastic Weight Consolidation (EWC)

EWC identifies parameters critical to previously learned tasks by approximating their posterior distribution using the Fisher Information Matrix. After training on Task A, the diagonal of the Fisher matrix quantifies how sensitive the loss is to small changes in each weight. Weights with high importance are penalized more heavily during subsequent training, effectively 'freezing' the knowledge in place while allowing less critical weights to adapt freely to new tasks.

The algorithm augments the standard loss function for a new task with a quadratic penalty term:

L(θ) = L_B(θ) + Σ_i (λ/2) * F_i * (θ_i - θ*_A,i)²

• L_B(θ): Loss for the new Task B
• λ: Hyperparameter controlling the strength of consolidation
• F_i: Fisher information for weight i from Task A
• θ*_A,i: Optimal weight value after learning Task A

This elastic penalty anchors important parameters to their previous values, proportional to their estimated significance.

Standard stochastic gradient descent overwrites weights indiscriminately, causing performance on old tasks to plummet. EWC provides a biologically inspired solution, mimicking the way mammalian brains protect consolidated memories through cascade synaptic plasticity. By reducing the learning rate on crucial synapses, the network can learn new patterns without erasing established ones, enabling a single model to maintain high accuracy across a sequence of disjoint tasks.

Unlike rehearsal-based methods that require storing or generating samples from previous tasks, EWC is a purely regularized approach with minimal memory overhead. It only stores two values per weight: the optimal parameter θ* and its Fisher information F. This makes it highly suitable for edge deployment and privacy-sensitive applications where retaining raw data from prior tasks is prohibited. The computational cost scales linearly with the number of parameters.

EWC excels in supervised continual learning benchmarks such as permuted MNIST, where the input pixels are shuffled differently for each task. A single network with EWC can learn 10 sequential permutations and achieve near-multi-task performance, whereas a plain SGD baseline suffers catastrophic degradation. This demonstrates its ability to partition the weight space into task-specific, non-overlapping sub-networks without explicit architectural growth.

The standard EWC makes a diagonal approximation of the Fisher matrix, ignoring off-diagonal weight interactions, which can lead to suboptimal consolidation for complex tasks. The hyperparameter λ also requires careful tuning to balance rigidity and plasticity. Modern extensions like Online EWC maintain a running estimate of importance, and Synaptic Intelligence (SI) offers an alternative importance measure based on the weight's contribution to loss reduction over its trajectory.

The primary problem EWC is designed to solve. When a neural network is trained sequentially on new tasks, its performance on previously learned tasks degrades rapidly as weights are overwritten. This occurs because standard gradient descent has no mechanism to protect parameters critical to earlier knowledge. Catastrophic forgetting is the fundamental barrier to building truly continual learning systems.

The mathematical backbone of EWC. The Fisher Information Matrix quantifies how sensitive the loss function is to changes in each network weight. In EWC, the diagonal of this matrix is computed after training on a task to identify which parameters are most important. These importance values then determine the strength of the quadratic penalty applied during subsequent training, effectively 'locking down' critical weights.

A complementary approach to EWC that computes weight importance online during training rather than post-hoc. Synaptic Intelligence tracks the contribution of each parameter to the reduction in loss over the entire learning trajectory. This path-based method can capture importance signals that point-estimate Fisher methods miss, particularly for tasks with complex loss landscapes.

An alternative importance estimation method that measures the sensitivity of the learned function's output to weight perturbations, rather than the loss. MAS computes importance by accumulating the magnitude of gradients of the model's output with respect to each parameter. This approach is unsupervised, requiring no labeled data to estimate weight importance, making it suitable for scenarios where task boundaries are unclear.

A structural rather than regularization-based approach to continual learning. Instead of penalizing weight changes, Progressive Neural Networks freeze previously learned columns and add new lateral connections for each new task. This completely eliminates forgetting but at the cost of growing model complexity linearly with the number of tasks. EWC offers a more parameter-efficient alternative.

A rehearsal-based strategy that interleaves samples from previous tasks during training on new ones. Unlike EWC's regularization approach, experience replay explicitly revisits old data to maintain performance. Hybrid methods combine both: using a small memory buffer alongside elastic weight consolidation to achieve stronger retention with minimal storage overhead.