Elastic Weight Consolidation (EWC)

EWC quantifies the importance of each network parameter to a learned task using the diagonal of the Fisher Information Matrix. This matrix estimates how much a change in a parameter would affect the model's output log-likelihood for the old task data. High Fisher values indicate parameters that are crucial for task performance; these are 'anchored' with a strong regularization penalty during new learning to prevent disruptive changes.

The core of EWC is a quadratic penalty term added to the loss function for learning a new task (Task B). For each parameter θᵢ, the penalty is: λ/2 * Fᵢ * (θᵢ - θ*ᵢ,A)², where:

• θ*ᵢ,A is the optimal parameter value after learning Task A.
• Fᵢ is its Fisher information (importance).
• λ is a hyperparameter scaling overall constraint strength. This creates an elastic constraint: important parameters (high Fᵢ) are confined to a tight region around their old values, while unimportant parameters can change more freely, allowing for new learning.

A defining feature of EWC is its exemplar-free operation. Unlike replay-based methods, it does not require storing raw data from previous tasks. After learning Task A, the algorithm stores only two small pieces of information per parameter: the optimal value (θ*A) and its estimated importance (F_A). This makes EWC highly memory-efficient and suitable for scenarios where data retention is prohibited due to privacy or storage constraints. The model's knowledge of Task A is distilled into this parameter-wise importance map.

EWC has a foundational Bayesian interpretation. It approximates the true posterior distribution over parameters p(θ|D_A) after learning Task A with a diagonal Gaussian distribution. The mean of this Gaussian is θ*A, and the precision (inverse variance) is given by the Fisher information F. The quadratic penalty term is derived from the log probability of this approximate posterior. Therefore, EWC effectively performs online Laplace approximation, treating important parameters as having low variance (high precision) in the posterior.

EWC introduces specific overheads:

• Memory: Must store a vector of Fisher values and optimal parameters for each learned task. For a model with N parameters and T tasks, storage is O(N*T).
• Computation: Calculating the diagonal Fisher information requires an additional pass over the Task A training data (or a representative subset) after training. The forward-backward pass computes gradients for the log-likelihood, which are then squared and averaged to estimate Fᵢ.
• Inference: No overhead; the consolidated model is a standard neural network.

While elegant, EWC has key limitations:

• Diagonal Approximation: Using only the diagonal of the Fisher Matrix ignores correlations between parameters, which can be significant in neural networks.
• Task Identity Requirement: Standard EWC requires knowledge of the task identity at test time to apply the correct set of constraints during training. Extensions are needed for pure task-agnostic learning.
• Hyperparameter Sensitivity: The constraint strength λ is critical. Too low leads to forgetting; too high stifles new learning (plasticity-stability trade-off).
• Long Task Sequences: Performance can degrade over many tasks as the Gaussian approximation becomes less accurate and the fixed λ may not be optimal for all tasks.

Catastrophic forgetting is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. It is the core problem that EWC and other continual learning algorithms aim to solve. This phenomenon occurs because gradient-based updates optimized for a new task are not constrained to preserve weights critical for old tasks, leading to unconstrained overwriting of previously acquired knowledge.

Learning without Forgetting is a seminal continual learning algorithm that, like EWC, avoids storing raw past data. Instead, it uses knowledge distillation from the model's own outputs. When learning a new task, LwF computes the model's predictions on new data before updating weights, then uses these 'old' predictions as soft targets in a distillation loss. This penalizes changes to the output distribution for the new data, encouraging the model to integrate new knowledge while preserving its existing response patterns.

Gradient Episodic Memory takes a constrained optimization approach to continual learning. It maintains a small episodic memory of real examples from past tasks. When computing a gradient update for a new task, GEM projects this new gradient so that it does not increase the loss on the examples in the episodic memory. This ensures updates are in a direction compatible with past knowledge. Unlike EWC's parameter-level regularization, GEM operates directly on the loss landscape via gradient constraints.

Experience replay is a core biological-inspired mechanism where a subset of past training data is stored in a memory buffer and replayed during the learning of new tasks. By intermittently retraining on a mixture of new data and old examples, the model rehearses previous knowledge. This directly combats forgetting but introduces memory overhead. Methods like Generative Replay use a generative model to create synthetic past data (pseudo-rehearsal), eliminating the need to store raw data but adding model complexity.

The plasticity-stability trade-off is the fundamental, opposing objectives in continual learning. Plasticity is a model's ability to learn new information and adapt to novel tasks. Stability is its ability to retain previously learned knowledge. All continual learning methods, including EWC, explicitly manage this trade-off:

• High plasticity, low stability: Leads to catastrophic forgetting.
• High stability, low plasticity: Leads to intransigence, where the model cannot learn new tasks. EWC addresses this by making important weights (for stability) elastic while allowing less important weights (for plasticity) to change more freely.

The Fisher Information Matrix is the mathematical cornerstone of EWC's importance estimation. For each parameter in a neural network, the Fisher information approximates how much that parameter contributes to the model's performance on a task. In practice, EWC uses the diagonal of this matrix, measuring the expected squared gradient of the log-likelihood. A high Fisher value for a weight indicates it is well-determined and crucial for task performance; changing it significantly would greatly increase the loss. This diagonal forms the basis for the quadratic penalty in the EWC loss function.