Continual Learning

Models learn from a non-stationary data stream where tasks or data distributions arrive one after another. Unlike batch learning, the model cannot access the entire dataset at once. This requires algorithms to process data in temporal order, simulating real-world deployment where information is received over time. A key challenge is that the data violates the independent and identically distributed (i.i.d.) assumption fundamental to standard training.

Systems operate under a strict memory budget, preventing the storage of all past data. This constraint is practical for edge devices and large-scale applications. Algorithms must be efficient, using strategies like:

• Experience Replay: Storing a small, fixed-size buffer of past examples.
• Generative Replay: Using a learned model to synthesize past data.
• Parameter Importance: Storing only importance scores for weights, not data. The goal is to approximate the benefit of full retraining using only a fraction of the original information.

This is the core tension in continual learning. A system must balance two opposing needs:

• Stability: Retaining knowledge from previous tasks (catastrophic forgetting is a failure of stability).
• Plasticity: Remaining flexible enough to learn new tasks effectively. Algorithms position themselves on this spectrum. Regularization-based methods (e.g., EWC) favor stability by constraining weight changes. Architectural methods (e.g., Progressive Networks) favor plasticity by adding new parameters, but at a cost of model growth.

In the most challenging and realistic setting (task-free or online class-incremental learning), the model does not receive a task identifier during inference. It must automatically recognize the context of an input and apply the appropriate knowledge without explicit guidance. This requires the model to develop an internal representation that disentangles or organizes features across all learned tasks, moving beyond simple multi-head output layers that rely on task IDs.

Performance is measured not just on the current task, but on how learning affects all tasks. Two key metrics define this:

• Backward Transfer (BWT): Measures the impact of learning a new task on performance of old tasks. Positive BWT indicates consolidation and improvement; negative BWT indicates forgetting.
• Forward Transfer (FWT): Measures how learning previous tasks improves performance on a new task, indicating useful knowledge reuse. The ideal continual learner exhibits strong positive transfer in both directions.

Standard average accuracy is insufficient. Continual learning requires specialized benchmarks and metrics:

• Learning Curves: Accuracy per task plotted over the entire training sequence.
• Forgetting Measure: Explicit calculation of performance drop on earlier tasks.
• Model Size & Efficiency: Tracking parameter growth and compute cost over time.
• Avalanche and ContinualAI provide standardized benchmarks (e.g., Split-MNIST, CORe50) to fairly compare algorithms across these multi-dimensional criteria.

Catastrophic forgetting is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new, non-i.i.d. data. It is the fundamental problem continual learning aims to solve.

• Mechanism: Occurs due to parameter interference, where gradient updates for a new task overwrite weights critical for old tasks.
• Contrast with Stability: The opposite of knowledge retention or stability.
• Example: A model trained to recognize cats, then dogs, may completely forget how to identify cats.

The stability-plasticity dilemma is the core trade-off in adaptive systems between retaining stable knowledge (stability) and integrating new information (plasticity). Continual learning algorithms explicitly manage this balance.

• Stability: The resistance to catastrophic forgetting.
• Plasticity: The capacity to learn new patterns quickly and efficiently.
• Engineering Goal: Design systems that are plastic enough to learn and stable enough to remember.

Experience Replay (ER) is a foundational replay-based method that stores a subset of past training examples in a fixed-size memory buffer and interleaves them with new data during training.

• Purpose: Approximates the i.i.d. data assumption by rehearsing old experiences.
• Key Challenge: Buffer management—strategies for which past examples to store and sample.
• Variant: Dark Experience Replay (DER) stores past model logits ('dark knowledge') in addition to data.

Elastic Weight Consolidation (EWC) is a seminal regularization-based method that adds a quadratic penalty to the loss function, constraining parameters important for previous tasks.

• Mechanism: Uses the diagonal Fisher Information Matrix to estimate parameter importance. Important weights are 'anchored' with a strong penalty.
• Analogy: Treats important synapses as springs with high elasticity.
• Limitation: Assumes a known task boundary to compute the Fisher.

Gradient Episodic Memory (GEM) is a replay-based method that stores past examples and uses gradient projection to ensure new task updates do not increase the loss on those memories.

• Core Operation: Solves a quadratic programming problem to project the proposed gradient into a region defined by constraints from past tasks.
• Guarantee: Theoretically ensures non-negative backward transfer (new learning doesn't harm old tasks).
• Practicality: More computationally intensive than simple rehearsal.

Task-Free Continual Learning is a challenging scenario where a model learns from a continuous, non-i.i.d. data stream without explicit task boundaries or identifiers provided during training or inference.

• Real-World Analogy: Closely mimics a real-world deployment where data distribution drifts smoothly.
• Increased Difficulty: Algorithms cannot rely on explicit task switches to trigger specific mechanisms (e.g., computing Fisher in EWC).
• Solution Approaches: Often requires online clustering, lifelong novelty detection, or self-supervised signals.