Regularization-Based Methods

The core mechanism is the addition of a regularization term to the standard training loss. This term penalizes changes to network parameters based on their estimated importance for prior tasks. The total loss becomes: L_total = L_new_task + λ * R(θ, θ_old, Ω), where R is the regularization function, Ω represents parameter importance scores, and λ controls the penalty strength.

These methods rely on estimating a per-parameter importance weight (Ω) after learning each task. Common estimators include:

• Fisher Information Matrix: Measures how sensitive the model's output distribution is to changes in a parameter (used in Elastic Weight Consolidation).
• Synaptic Intelligence: Computes the cumulative gradient-based contribution of each parameter to the decrease in loss over past tasks.
• Magnitude-Based: Simpler heuristics where importance is proportional to the final parameter value or its change during training.

Regularization terms differ in how they penalize deviation from old parameters (θ_old).

• Quadratic Penalty: Used in methods like Elastic Weight Consolidation (EWC). The penalty is Σ_i Ω_i * (θ_i - θ_old_i)². This strongly anchors important parameters close to their old values.
• Linear/Sparse Penalty: Used in methods like Synaptic Intelligence (SI). The penalty is Σ_i Ω_i * |θ_i - θ_old_i|. This can encourage more sparse updates and different optimization dynamics.

A key advantage for edge deployment is that most regularization-based methods do not require storing raw data from previous tasks. Instead, they retain only a small amount of metadata: the old parameters (θ_old) and the importance matrix (Ω). This makes them highly suitable for devices with severe storage constraints, unlike rehearsal-based methods which need a replay buffer.

The primary compute cost is in calculating the importance weights Ω after each task, which typically requires an additional backward pass over the task data. During training on a new task, the regularization term adds minimal overhead to the forward/backward pass—often just an extra element-wise multiplication and addition. This predictable, low overhead is desirable for on-device training cycles.

These methods explicitly manage the stability-plasticity dilemma. The regularization strength λ is the critical hyperparameter:

• High λ: High stability, strong protection of old knowledge, but can stifle plasticity and slow learning of new tasks.
• Low λ: High plasticity, fast adaptation to new data, but increased risk of catastrophic forgetting. Tuning λ is essential for balancing performance across all learned tasks.

Regularization-based methods share core attributes and inherent limitations:

• Parameter Efficiency: They modify the loss function without adding model parameters, unlike architectural methods.
• No Raw Data Storage: Typically, they do not require storing past input data (except hybrids like RWalk), aiding privacy and memory efficiency.
• Importance Estimation: A core challenge is accurately and efficiently estimating parameter importance (via Fisher, path integral, or output sensitivity).
• Task Inference: Most assume knowledge of the task identity during training and sometimes during inference.
• Capacity Saturation: As tasks accumulate, the constraints can severely limit plasticity, leading to a stability-plasticity dilemma. The fixed network may eventually run out of free parameters to learn new tasks effectively.
• Computational Overhead: Calculating importance matrices (e.g., Fisher) adds pre- or post-task computation, though less than full rehearsal.

Catastrophic Forgetting is the phenomenon where a neural network abruptly and drastically loses previously learned information when trained on new data. It is the core problem that continual learning methods, including regularization, aim to solve.

• Mechanism: Occurs due to unconstrained parameter overwriting; new task gradients shift weights optimized for old tasks.
• Analogy: Like a student mastering calculus but then completely forgetting basic algebra after a new physics course.
• Impact: Renders sequential training impractical for real-world systems that must adapt over time.

Elastic Weight Consolidation is a foundational regularization-based method. It estimates the importance (Fisher information) of each network parameter for previous tasks and applies a quadratic penalty to discourage changes to important weights.

• Key Idea: Treats neural network parameters as springs; important weights have high elasticity (resistance to change).
• Loss Function: Adds a term: λ/2 * Σ_i F_i (θ_i - θ_i)^2, where F_i is the importance, θ_i is the old parameter value, and λ is a regularization strength.
• Use Case: Effective for task-incremental scenarios where task identity is known, but can struggle with strict online or class-incremental settings due to fixed importance estimates.

Synaptic Intelligence is an online, regularization-based method that computes parameter importance during training. It accumulates the contribution of each weight to the reduction in loss over time.

• Online Importance: Importance ω for a parameter is the path integral of the gradient ⋅ parameter change: ω = Σ_t (grad ⋅ Δθ).
• Dynamic Penalty: Applies a loss penalty proportional to ω * (Δθ)^2, protecting important synapses in real-time.
• Advantage over EWC: Does not require a separate Fisher information calculation phase after each task, making it more suitable for online continual learning on edge devices.

The Stability-Plasticity Dilemma is the fundamental trade-off in continual learning between a model's stability (retaining old knowledge) and its plasticity (efficiently learning new information).

• Stability: Achieved by methods like regularization (penalizing change) or parameter isolation (freezing weights). High stability risks intransigence (inability to learn new tasks).
• Plasticity: Achieved by allowing significant parameter updates. High plasticity leads to catastrophic forgetting.
• Engineering Goal: All continual learning algorithms, including regularization-based ones, seek an optimal operating point on this spectrum for a given application.

Rehearsal-Based Methods are a primary alternative to regularization. They retain a subset of past data in a replay buffer and interleave it with new data during training.

• Core Mechanism: Directly rehearses old tasks by retraining on stored examples (experience replay) or generated samples (generative replay).
• Contrast with Regularization: Addresses forgetting by re-exposing the model to old data, rather than just penalizing parameter change.
• Edge Challenge: Maintaining a buffer consumes memory. Buffer management strategies (e.g., reservoir sampling, coreset selection) are critical for edge deployment.

Backward Transfer is a key evaluation metric for continual learning that measures the impact (positive or negative) that learning a new task has on the performance of previously learned tasks.

• Positive Backward Transfer (BWT > 0): Learning Task B improves performance on Task A. This is rare but desirable.
• Negative Backward Transfer (BWT < 0): Learning Task B harms performance on Task A. This is catastrophic forgetting.
• Regularization's Goal: Methods like EWC and SI aim to maximize BWT (bring it close to zero) by minimizing negative interference. It is quantitatively measured as the average performance change on all previous tasks after learning new ones.