Continual Learning

These techniques add a penalty term to the loss function to constrain weight updates, protecting parameters deemed important for previous tasks. The core idea is to slow down learning on weights that were critical for past performance.

• Elastic Weight Consolidation (EWC): Calculates a Fisher Information Matrix to estimate the importance of each network parameter for a learned task. The loss function includes a quadratic penalty that prevents important weights from changing significantly.
• Synaptic Intelligence (SI): Online method that accumulates a per-parameter importance measure throughout training, based on the cumulative gradient updates. This allows importance to be estimated without storing old task data.
• Learning without Forgetting (LwF): Uses knowledge distillation by generating 'soft labels' from the old model's output on new data, encouraging the new model to maintain its original responses while learning the new task.

These approaches dynamically expand or partition the neural network model itself to allocate dedicated capacity for new tasks, physically isolating parameters to prevent interference.

• Progressive Neural Networks: Introduce entirely new columns of parameters for each new task. Lateral connections from previous columns allow the new column to leverage prior features, but old columns remain frozen, guaranteeing no forgetting.
• Dynamic Architecture / Parameter Expansion: Methods like PackNet or Piggyback learn binary masks to select a subset of weights for each task. The network capacity grows as tasks are added, but inference uses only the masked subset for a given task.
• Expert Networks (e.g., HAT): Use task-specific attention mechanisms or gating functions over a shared base network, activating different pathways. The Hard Attention to Task (HAT) mechanism learns binary attention masks to completely shut off gradients to parts of the network for certain tasks.

These strategies retain a subset of data from previous tasks (or generate synthetic examples) and interleave them with new task data during training. This directly re-exposes the model to old patterns.

• Experience Replay: Stores a fixed-size memory buffer of raw data samples from past tasks. During training on a new task, it samples a mini-batch containing a mix of new data and old replay data.
• Generative Replay: Trains a generative model (e.g., a Generative Adversarial Network or Variational Autoencoder) on the data distribution of each task. Instead of storing raw data, it stores the generator, which can produce pseudo-samples of old tasks to interleave with new real data.
• iCaRL (Incremental Classifier and Representation Learning): Combines replay with a prototype-based classification rule. It stores a small number of exemplars per class and uses a nearest-mean-of-exemplars classifier, which is more stable than a softmax layer when the number of classes grows.

A subset of architectural methods focused on identifying and freezing task-specific sub-networks within a larger, shared parameter space, enabling efficient inference.

• Sparse Coding / Supermasks: Methods like Lottery Ticket Hypothesis adaptations find sparse, trainable sub-networks (winning tickets) within a larger network that are sufficient for a given task. Only these sub-networks are activated and updated per task.
• Diffusion-based Mask Learning: Learns soft, differentiable masks for each task that can be applied to a shared backbone. The masks determine which neurons contribute to which task's output, allowing for some overlap and knowledge transfer while minimizing conflict.
• Context-Dependent Gating: Uses a context vector (e.g., a task ID embedding) to generate a gating signal that modulates neuron activations, effectively creating dynamic, task-specific network pathways without physical parameter duplication.

These approaches frame continual learning itself as a meta-problem. The goal is to learn an update rule or model initialization that is inherently robust to sequential task learning.

• Model-Agnostic Meta-Learning (MAML) for CL: Seeks a good initial set of parameters such that a few gradient steps on a new task lead to strong performance without harming performance on old tasks. The meta-objective explicitly trains for fast adaptation with minimal interference.
• Online Aware Meta-Learning (OML): Introduces an information-theoretic objective during meta-training that encourages the learned representations to be maximally reusable for future tasks while being minimally affected by gradient updates, promoting stability.
• Meta-Experience Replay (MER): Applies a meta-learning objective directly to the experience replay process, optimizing not just for task performance but for how the replay strategy itself affects the trade-off between learning the new task and remembering the old one.

These techniques leverage probabilistic deep learning to maintain a distribution over model parameters, naturally quantifying uncertainty about old and new data to guide stable learning.

• Bayesian Continual Learning: Treats model parameters as probability distributions (Bayesian Neural Networks). When learning a new task, the posterior from the previous task becomes the prior. This formally balances old and new evidence but is computationally challenging.
• Uncertainty-Guided Regularization: Uses estimates of epistemic uncertainty (from methods like Monte Carlo Dropout or ensembles) to identify parameters the model is uncertain about. Regularization is applied more strongly to parameters with low uncertainty (confident about old tasks) to protect them.
• Variational Continual Learning (VCL): A practical approximation to full Bayesian CL. It uses variational inference to maintain a Gaussian approximation to the posterior over weights after each task. The loss includes the Kullback-Leibler (KL) divergence between the new variational posterior and the old one, preventing drastic shifts.

Catastrophic forgetting is the tendency of a neural network to abruptly and completely lose previously learned information when trained on new tasks or data distributions. It is the primary challenge continual learning aims to solve.

• Mechanism: Occurs due to overwriting of shared weights critical for old tasks when optimizing for new ones.
• Example: A model trained to classify cats, then dogs, may forget how to recognize cats entirely.
• Mitigation: Techniques include elastic weight consolidation, experience replay, and progressive neural networks.

Experience replay is a technique where an agent stores past experiences (state, action, reward, next state) in a memory buffer and samples from it during training. It is a foundational method for mitigating catastrophic forgetting in continual and reinforcement learning.

• Function: Breaks temporal correlations in sequential data and provides a mechanism for rehearsal of old tasks.
• Implementation: Often uses a fixed-size replay buffer that stores a subset of past data, which is interleaved with new task data during training.
• Variants: Prioritized experience replay samples important transitions more frequently to accelerate learning.

Elastic Weight Consolidation (EWC) is a regularization-based continual learning algorithm that slows down learning on network weights identified as important for previous tasks.

• Core Idea: Calculates a Fisher information matrix to estimate the importance of each parameter to a learned task. Important weights are then "anchored" with a quadratic penalty.
• Analogy: Acts like a spring attached to each important weight, applying a restoring force if training on a new task tries to move it too far.
• Use Case: Particularly effective for task-incremental learning where task boundaries are known.

Meta-learning (or 'learning to learn') is a framework where a model is trained on a wide distribution of tasks so it can rapidly adapt to new, unseen tasks with minimal data. It is a complementary approach to continual learning.

• Objective: Learns a general-purpose initialization or learning algorithm that is highly adaptable.
• Relation to CL: Meta-learning can provide a strong, flexible starting point (pre-adaptation) that makes subsequent continual learning more efficient and stable.
• Algorithms: Includes Model-Agnostic Meta-Learning (MAML) and Reptile, which optimize for fast adaptation.

Progressive Neural Networks are an architectural approach to continual learning that avoids catastrophic forgetting by instantiating a new neural network column for each new task, while allowing information flow from previous columns via lateral connections.

• Mechanism: Freezes the parameters of columns for old tasks, guaranteeing no forgetting. New tasks benefit from features extracted by previous columns.
• Advantage: Provides a strong upper bound on retaining old knowledge.
• Drawback: Leads to linear growth in parameters with the number of tasks, which can become computationally expensive.

Rehearsal-based methods are a class of continual learning techniques that retain a subset of data from previous tasks (a rehearsal buffer) and replay it alongside data from the current task during training.

• Principle: Directly addresses forgetting by providing interleaved exposure to old and new data distributions.
• Implementation: Can use raw data storage, stored feature representations, or generative models to produce pseudo-samples of old data.
• Challenge: Requires managing memory constraints and potential privacy concerns when storing raw data.