Elastic Weight Consolidation

EWC adds a quadratic penalty to the loss function that anchors important parameters to their optimal values from previous tasks. The penalty is proportional to the Fisher Information Matrix diagonal, which quantifies how sensitive the loss is to changes in each weight. Weights with high Fisher values are 'stiff' and resist change, while low-value weights remain 'elastic' for new learning.

The Fisher Information Matrix serves as EWC's importance estimator. It measures how much the model's output distribution changes when a specific weight is perturbed. Computed after training on a task, the diagonal approximation identifies which parameters encode core knowledge. This avoids the storage overhead of retaining previous task data while providing a principled, second-order approximation of weight importance.

EWC is directly inspired by neuroscience models of synaptic consolidation. In mammalian brains, the neocortex consolidates memories by reducing the plasticity of synapses that store crucial knowledge. EWC mimics this computationally:

• Protected weights: Analogous to consolidated synapses with low plasticity
• Elastic weights: Analogous to unconsolidated synapses free to encode new information
• Task sequence: Mirrors how biological systems learn sequentially without erasing prior skills

Standard neural networks suffer from catastrophic forgetting—new task training overwrites weights optimized for prior tasks. EWC solves this by constraining gradient updates in directions that damage historical performance. The algorithm finds a parameter subspace where new tasks can be learned with minimal interference, enabling a single model to maintain high accuracy across sequential tasks without data rehearsal or architectural expansion.

EWC operates in three distinct phases per task:

1. Train on Task A: Standard gradient descent to convergence
2. Compute Fisher Diagonal: Estimate weight importance using post-training gradients
3. Consolidate & Proceed: Add EWC penalty to loss function before training on Task B

The penalty term is λ/2 * Σ F_i * (θ_i - θ*_A,i)², where F_i is the Fisher diagonal, θ_i are current weights, and θ*_A,i are the optimal weights for Task A. The hyperparameter λ controls consolidation strength.

While foundational, vanilla EWC has constraints that subsequent research addresses:

• Diagonal approximation: Ignores off-diagonal Fisher entries, missing weight correlations
• Fixed importance: Assumes static weight importance, which may drift over long task sequences
• Memory scaling: Requires storing one set of optimal weights and Fisher values per task
• Extensions: Online EWC maintains a single running Fisher estimate; Synaptic Intelligence uses a path-integral importance measure; Memory Aware Synapses computes importance in an unsupervised manner

EWC works by adding a quadratic penalty term to the standard loss function. This term penalizes changes to weights deemed important for previous tasks. The importance is quantified by the Fisher Information Matrix, which estimates the sensitivity of the model's output to small changes in each weight. The higher the Fisher information for a weight, the more rigid it becomes, effectively slowing down learning on those critical parameters and preserving old knowledge.

The primary problem EWC is designed to solve. It is the tendency of a neural network to abruptly and completely overwrite previously learned knowledge upon training on a new task. This occurs because standard gradient descent optimizes all weights for the current objective without any mechanism to protect prior representations. EWC provides a biologically-inspired solution, mimicking the synaptic consolidation observed in the mammalian neocortex.

A core mathematical object in EWC, the Fisher Information Matrix (FIM) serves as a proxy for weight importance. It measures the amount of information an observable variable carries about a model parameter. In EWC, the diagonal of the FIM is computed after training on a task. A high value indicates that a small perturbation to that weight would significantly change the model's output, marking it as crucial and thus heavily constrained during future learning.

A related continual learning algorithm that, unlike EWC, computes weight importance online during the training trajectory. It tracks the contribution of each parameter to the reduction of the loss over time. This path-integral approach allows Synaptic Intelligence (SI) to consolidate knowledge without needing a separate post-training phase to compute the Fisher Information Matrix, making it more adaptable in streaming data scenarios.

Memory Aware Synapses (MAS) is an alternative to EWC that estimates parameter importance based on the sensitivity of the learned function's output, rather than the loss function. It measures the magnitude of the gradient of the squared L2-norm of the model's output with respect to each parameter. This makes MAS a task-agnostic method, as it does not require access to the loss function or labeled data from previous tasks to compute importance weights.

A different family of continual learning strategies that explicitly store and replay data from previous tasks. Experience Replay interleaves old examples with new data during training, while Generative Replay trains a generative model to synthesize pseudo-samples of past tasks. These methods directly counteract forgetting by maintaining a form of memory, but they incur storage or computational overhead that EWC's regularized approach avoids.