Elastic Weight Consolidation (EWC)

EWC quantifies parameter importance using the Fisher Information Matrix. For a parameter (\theta_i), the diagonal Fisher information (F_i) is estimated as the expected square of the gradient of the log-likelihood with respect to that parameter. A high (F_i) value indicates that changing (\theta_i) would cause a large change in the model's output distribution for the previous task, marking it as important and requiring consolidation.

• Calculation: (F_i = \mathbb{E}{x \sim D{old}}[(\frac{\partial \log p(y|x, \theta)}{\partial \theta_i})^2])
• Interpretation: This approximates the curvature (second derivative) of the loss landscape around the optimal parameters for the old task.

The EWC loss function adds a regularization term to the standard cross-entropy loss for the new task. This term is a quadratic penalty that constrains each parameter (\theta_i) to stay close to its old optimal value (\theta_{i, A}^*), weighted by its importance (F_i).

• Formula: (L(\theta) = L_{B}(\theta) + \frac{\lambda}{2} \sum_i F_i (\theta_i - \theta_{i, A}^*)^2)
• Effect: The model can still learn the new task (B) but must pay a "cost" proportional to (F_i) for moving important parameters away from their values for task A. The hyperparameter (\lambda) controls the strength of consolidation.

Computing and storing the full Fisher Information Matrix is intractable for large models. EWC makes two key approximations for feasibility:

• Diagonal Approximation: Only the diagonal elements of the Fisher matrix are stored, assuming parameters are independent. This reduces storage from (O(n^2)) to (O(n)).
• Point Estimate: The Fisher is often computed at the optimal parameters (\theta_A^*) after training on the first task, rather than averaged over the entire data distribution.

These approximations make EWC computationally lightweight compared to other second-order methods, a critical feature for edge deployment.

EWC can be applied sequentially to a stream of tasks. After learning task (k), the algorithm:

1. Records the optimal parameters (\theta_k^*) for that task.
2. Estimates the Fisher Information (F^{(k)}) for task (k).
3. When learning task (k+1), the loss function becomes a sum of penalties for all previous tasks: (L(\theta) = L_{k+1}(\theta) + \sum_{j=1}^{k} \frac{\lambda}{2} \sum_i F_i^{(j)} (\theta_i - \theta_{i, j}^*)^2).

This creates an elastic consolidation of parameters important for all past tasks, forming a complex, multi-task constraint landscape.

EWC has a theoretical foundation in Bayesian inference. Learning task B after task A is framed as computing the posterior (P(\theta|D_A, D_B)). Under a Laplace approximation, assuming the posterior for task A is a Gaussian distribution with mean (\theta_A^*) and a diagonal precision matrix given by the Fisher Information (F), the EWC loss emerges as the log posterior. Therefore, EWC approximates sequential Bayesian updating, treating important parameters (high Fisher) as having high prior precision.

Strengths for Edge-CL:

• Minimal Memory Overhead: Only requires storing a vector of Fisher values and old parameters per task ((O(n))), not raw data.
• Computational Efficiency: The quadratic penalty adds negligible cost during training.
• Online Fisher Estimation: Can be approximated online during the initial task training.

Key Limitations:

• Diagonal Assumption: Ignores parameter correlations, which can limit effectiveness.
• Quadratic Constraint: May overly restrict plasticity if the Fisher estimate is inaccurate.
• Task Identity Required: Typically assumes knowledge of the task boundary to compute and apply the Fisher, though variants exist for task-agnostic settings.

EWC's core mechanism requires storing a Fisher Information Matrix (FIM) for each previous task, which is a diagonal matrix with the same dimensionality as the model's parameters. For a model with N parameters, this imposes an O(N) memory overhead per task. On memory-constrained edge devices, this can quickly become prohibitive for long task sequences or large models, necessitating approximations like storing only the top-K most important weights or using a moving average of a single importance matrix.

The effectiveness of EWC hinges on the regularization strength hyperparameter (λ). This scalar balances the penalty for changing important old weights against the need to learn the new task.

• Too high λ: The model becomes overly rigid (underfitting on the new task).
• Too low λ: Catastrophic forgetting reoccurs. Tuning λ is non-trivial, as the optimal value depends on task similarity, model architecture, and data distribution. It often requires expensive cross-validation, which is impractical in true online edge learning scenarios.

EWC's mathematical derivation assumes the loss landscape for a previous task is locally quadratic around the learned parameters. This allows it to approximate the importance of weights via the diagonal of the FIM (the second derivative). In reality, neural network loss landscapes are highly non-convex and may not be well-approximated by a simple quadratic. This assumption can break down, especially after multiple sequential tasks, leading to suboptimal importance estimates and reduced effectiveness in preventing forgetting.

Accurately computing the Fisher Information Matrix requires a forward-backward pass over the entire dataset of the learned task. For edge devices learning from streaming data, this is a significant bottleneck. Common approximations include:

• Online Fisher Estimation: Updating the FIM with a running average during the initial task training.
• Monte Carlo Sampling: Using a small, random subset of the data or model outputs. These approximations reduce accuracy but are essential for feasibility. The cost scales with model size and must be factored into the device's energy and thermal budget.

EWC is a regularization-based method that creates a single, shared set of parameters for all tasks. Unlike architectural methods (e.g., Progressive Neural Networks), it does not provide an explicit mechanism to identify which task a given input belongs to at inference time. Therefore, EWC is primarily suited for Task-Incremental Learning scenarios where a task ID is provided at test time. For Class-Incremental Learning on edge devices—where the model must distinguish between all seen classes without a task cue—EWC must be combined with other techniques like a replay buffer or a dynamic output head.

As the number of tasks grows, the sum of quadratic penalties in the EWC loss function accumulates. This can lead to over-regularization, where the model's plasticity is severely limited for learning new tasks because almost all parameters have accrued some importance from past tasks. The model's capacity can become saturated. Mitigation strategies include:

• Pruning unimportant weights from the importance matrix.
• Using a single, consolidated importance measure that merges information from all past tasks rather than summing individual penalties.
• Transitioning to a hybrid approach that combines EWC with a small replay buffer.

Catastrophic Forgetting is the phenomenon where a neural network abruptly and drastically loses previously learned information when trained on new data. It is the core problem EWC and other continual learning methods aim to solve.

• Occurs due to unconstrained parameter overwriting during gradient descent.
• Represents the plasticity side of the stability-plasticity dilemma.
• Example: A model trained to classify cats vs. dogs, when later trained to classify cars, loses its ability to distinguish pets.

Synaptic Intelligence (SI) is a closely related regularization-based continual learning algorithm. Like EWC, it estimates parameter importance to mitigate forgetting, but it calculates importance online during training.

• Key Difference from EWC: SI computes importance as an integral over the parameter's optimization path, while EWC uses the Fisher Information Matrix computed after a task is complete.
• This allows SI to operate in a more flexible, online continual learning setting without needing a separate importance estimation phase.

Regularization-Based Methods are a family of continual learning techniques that add a penalty term to the loss function to protect knowledge from previous tasks. EWC is a premier example.

• The penalty discourages changes to parameters deemed important for old tasks.
• Pros: Memory-efficient, as they do not require storing raw past data.
• Cons: Can struggle with long task sequences due to parameter rigidity and potential miscalibration of importance.
• Other examples include Learning without Forgetting (LwF) and Memory Aware Synapses (MAS).

The Stability-Plasticity Dilemma is the fundamental trade-off in continual learning and neural networks. EWC explicitly manages this trade-off.

• Stability: The ability to retain consolidated knowledge (old tasks). EWC promotes stability via the quadratic penalty.
• Plasticity: The ability to integrate new knowledge (new tasks). EWC allows plasticity for parameters with low Fisher information.
• All continual learning algorithms position themselves differently on this spectrum. EWC uses a parameter-wise granularity to balance the two.

The Fisher Information Matrix (FIM) is a mathematical construct from statistics that EWC uses to measure parameter importance. It quantifies how much information an observable random variable (model outputs) carries about an unknown parameter (model weights).

• In EWC, the diagonal of the FIM is used. A high Fisher value for a weight indicates that changing it would significantly change the model's output distribution for the previous task.
• This provides a local, task-specific measure of importance, justifying a strong penalty to preserve that weight's value.

Class-Incremental Learning (CIL) is a primary evaluation scenario for EWC and other continual learning methods. The model must learn new classes sequentially without forgetting old ones, and perform inference without being told the task identity.

• This is a strict and challenging setting, as the model must distinguish between all seen classes.
• EWC alone often struggles with CIL; it is frequently combined with a rehearsal buffer or a dynamic architecture component.
• Contrast with Task-Incremental Learning, where the task ID is provided at test time, simplifying the problem.