Elastic Weight Consolidation (EWC)

EWC's core mechanism is a quadratic penalty term added to the loss function. This term penalizes changes to parameters based on their estimated importance for previous tasks. The importance is quantified by the diagonal of the Fisher Information Matrix, which approximates how much each parameter contributes to the model's performance on a given task. The loss function becomes: L_new(θ) = L(θ) + λ/2 * Σ_i F_i * (θ_i - θ*_i)^2, where θ* are the optimal parameters from the old task, F_i is the Fisher importance, and λ is a hyperparameter controlling the strength of consolidation.

EWC requires estimating which parameters are 'important' and which are 'plastic'. This is done by computing the Fisher Information for each parameter after training on a task. In practice, this is often approximated by the empirical Fisher: the average squared gradient of the log-likelihood with respect to the model's output. A high Fisher value for a weight indicates that perturbing it significantly increases the loss, marking it as crucial for the task. This estimation is a post-hoc analysis performed once per task and stored alongside the optimal parameters.

The penalty term imposes a quadratic constraint around the old optimal parameters. This creates an 'elastic' restoring force: the further a parameter moves from its old value, the stronger the penalty, but small movements are allowed. This elasticity is key to the stability-plasticity dilemma. Important parameters (high Fisher) have a stiff spring, resisting change. Unimportant parameters (low Fisher) have a weak spring, allowing plasticity for learning new tasks. This selective rigidity enables sequential learning without catastrophic interference.

EWC has two primary formulations:

• Task-Aware EWC: The standard formulation assumes explicit task boundaries. A separate Fisher matrix and parameter anchor (θ*) are stored for each previous task. The total penalty is the sum of constraints from all past tasks.
• Task-Agnostic / Online EWC: Designed for scenarios without clear task boundaries. It maintains a running estimate of a single, cumulative Fisher matrix and a single moving parameter anchor, allowing for continual adaptation in a non-stationary stream. This is more practical for real-world deployment but can be less precise.

EWC introduces specific overheads compared to standard training:

• Memory: Must store the optimal parameters (θ*) and the diagonal Fisher matrix (F) for each learned task. For a model with N parameters, this requires O(N) storage per task.
• Computation: The Fisher matrix estimation requires an additional forward/backward pass over the training data after convergence. The quadratic penalty term adds minimal cost during subsequent training. The overhead is generally lower than replay-based methods that store raw data, but scales with the number of parameters, not tasks.

While foundational, EWC has known limitations:

• Quadratic Approximation: The Fisher diagonal is a local, second-order approximation. It may fail for highly non-convex loss landscapes or when new tasks induce large, necessary parameter shifts.
• Parameter Corruption: In sequential multi-task scenarios, the Fisher estimates for early tasks can become inaccurate after many subsequent updates, reducing their protective efficacy.
• Hyperparameter Sensitivity: The performance is highly sensitive to the choice of the regularization strength λ. Setting it requires cross-validation, which is challenging in true continual learning.
• Assumption of Fixed Architecture: Like most regularization methods, EWC cannot expand model capacity to accommodate fundamentally new features.

Catastrophic forgetting is the core problem EWC addresses. It is the tendency of an artificial neural network to abruptly and drastically lose performance on previously learned tasks when trained on new, non-i.i.d. data. This occurs because gradient-based optimization overwrites parameters that were critical for old tasks, as the network's plasticity for new learning conflicts with the stability of existing knowledge. It is the primary obstacle to building models that learn continuously over time.

Synaptic Intelligence (SI) is a closely related regularization-based algorithm. Like EWC, it adds a penalty to the loss function to protect important parameters. However, SI calculates parameter importance online during the initial training on each task, tracking the cumulative gradient updates for each weight. This results in an importance measure based on the actual trajectory of learning, whereas EWC's Fisher information is a post-hoc, offline calculation after a task is learned. Both demonstrate the power of parameter-specific regularization.

The Fisher Information Matrix (FIM) is the mathematical foundation of EWC's importance measure. For a model parameter, the Fisher information estimates how much the model's output distribution changes with a small perturbation of that parameter. In EWC, the diagonal of the FIM is computed on the data from a learned task. A high Fisher value for a weight indicates it is highly informative for that task; changing it would significantly alter the model's predictions. EWC uses these values to define the strength of its quadratic penalty.

This is the overarching category of continual learning algorithms to which EWC belongs. Regularization-based methods mitigate forgetting by adding an auxiliary penalty term to the standard loss function when learning a new task. This term constrains the optimization, discouraging changes to parameters deemed important for previous tasks. Key variants include:

• EWC/SI: Use parameter importance.
• Learning without Forgetting (LwF): Uses knowledge distillation loss on new task data. Their advantage is fixed model size and no need to store raw data, but they can struggle with many sequential tasks.

Experience Replay (ER) represents the primary alternative paradigm to regularization-based methods like EWC. Instead of constraining weights, ER stores a subset of past training examples in a fixed memory buffer. During training on a new task, it interleaves these old examples (replays) with new data. This directly approximates the i.i.d. assumption of standard batch training. While EWC is parameter-centric, ER is data-centric. Hybrid approaches often combine replay with distillation or regularization for stronger performance.

Backward Transfer (BWT) is a critical metric for evaluating continual learning algorithms like EWC. It measures the influence of learning a new task on the performance of all previously learned tasks. It is calculated as the average change in accuracy on all old tasks after learning the new task.

• Negative BWT: Indicates catastrophic forgetting (performance decreased).
• Zero BWT: Indicates perfect stability (no forgetting).
• Positive BWT: Indicates consolidation (new learning improved old tasks). A primary goal of EWC is to maximize BWT, or at least keep it close to zero.