Continual Learning

These strategies retain a subset of data from previous tasks (or generate synthetic examples) and interleave them with new task data during training. This directly re-exposes the model to old patterns.

• Experience Replay: Stores a fixed-size memory buffer of raw data samples from past tasks. During training on a new task, it samples a mini-batch containing a mix of new data and old replay data.
• Generative Replay: Trains a generative model (e.g., a Generative Adversarial Network or Variational Autoencoder) on the data distribution of each task. Instead of storing raw data, it stores the generator, which can produce pseudo-samples of old tasks to interleave with new real data.
• iCaRL (Incremental Classifier and Representation Learning): Combines replay with a prototype-based classification rule. It stores a small number of exemplars per class and uses a nearest-mean-of-exemplars classifier, which is more stable than a softmax layer when the number of classes grows.

A subset of architectural methods focused on identifying and freezing task-specific sub-networks within a larger, shared parameter space, enabling efficient inference.

• Sparse Coding / Supermasks: Methods like Lottery Ticket Hypothesis adaptations find sparse, trainable sub-networks (winning tickets) within a larger network that are sufficient for a given task. Only these sub-networks are activated and updated per task.
• Diffusion-based Mask Learning: Learns soft, differentiable masks for each task that can be applied to a shared backbone. The masks determine which neurons contribute to which task's output, allowing for some overlap and knowledge transfer while minimizing conflict.
• Context-Dependent Gating: Uses a context vector (e.g., a task ID embedding) to generate a gating signal that modulates neuron activations, effectively creating dynamic, task-specific network pathways without physical parameter duplication.

These approaches frame continual learning itself as a meta-problem. The goal is to learn an update rule or model initialization that is inherently robust to sequential task learning.

• Model-Agnostic Meta-Learning (MAML) for CL: Seeks a good initial set of parameters such that a few gradient steps on a new task lead to strong performance without harming performance on old tasks. The meta-objective explicitly trains for fast adaptation with minimal interference.
• Online Aware Meta-Learning (OML): Introduces an information-theoretic objective during meta-training that encourages the learned representations to be maximally reusable for future tasks while being minimally affected by gradient updates, promoting stability.
• Meta-Experience Replay (MER): Applies a meta-learning objective directly to the experience replay process, optimizing not just for task performance but for how the replay strategy itself affects the trade-off between learning the new task and remembering the old one.

These techniques leverage probabilistic deep learning to maintain a distribution over model parameters, naturally quantifying uncertainty about old and new data to guide stable learning.

• Bayesian Continual Learning: Treats model parameters as probability distributions (Bayesian Neural Networks). When learning a new task, the posterior from the previous task becomes the prior. This formally balances old and new evidence but is computationally challenging.
• Uncertainty-Guided Regularization: Uses estimates of epistemic uncertainty (from methods like Monte Carlo Dropout or ensembles) to identify parameters the model is uncertain about. Regularization is applied more strongly to parameters with low uncertainty (confident about old tasks) to protect them.
• Variational Continual Learning (VCL): A practical approximation to full Bayesian CL. It uses variational inference to maintain a Gaussian approximation to the posterior over weights after each task. The loss includes the Kullback-Leibler (KL) divergence between the new variational posterior and the old one, preventing drastic shifts.

Elastic Weight Consolidation (EWC) is a regularization-based continual learning algorithm that slows down learning on network weights identified as important for previous tasks.

• Core Idea: Calculates a Fisher information matrix to estimate the importance of each parameter to a learned task. Important weights are then "anchored" with a quadratic penalty.
• Analogy: Acts like a spring attached to each important weight, applying a restoring force if training on a new task tries to move it too far.
• Use Case: Particularly effective for task-incremental learning where task boundaries are known.

Meta-learning (or 'learning to learn') is a framework where a model is trained on a wide distribution of tasks so it can rapidly adapt to new, unseen tasks with minimal data. It is a complementary approach to continual learning.

• Objective: Learns a general-purpose initialization or learning algorithm that is highly adaptable.
• Relation to CL: Meta-learning can provide a strong, flexible starting point (pre-adaptation) that makes subsequent continual learning more efficient and stable.
• Algorithms: Includes Model-Agnostic Meta-Learning (MAML) and Reptile, which optimize for fast adaptation.

Progressive Neural Networks are an architectural approach to continual learning that avoids catastrophic forgetting by instantiating a new neural network column for each new task, while allowing information flow from previous columns via lateral connections.

• Mechanism: Freezes the parameters of columns for old tasks, guaranteeing no forgetting. New tasks benefit from features extracted by previous columns.
• Advantage: Provides a strong upper bound on retaining old knowledge.
• Drawback: Leads to linear growth in parameters with the number of tasks, which can become computationally expensive.

Rehearsal-based methods are a class of continual learning techniques that retain a subset of data from previous tasks (a rehearsal buffer) and replay it alongside data from the current task during training.

• Principle: Directly addresses forgetting by providing interleaved exposure to old and new data distributions.
• Implementation: Can use raw data storage, stored feature representations, or generative models to produce pseudo-samples of old data.
• Challenge: Requires managing memory constraints and potential privacy concerns when storing raw data.