Catastrophic Forgetting

Catastrophic forgetting occurs when the fundamental assumption of independent and identically distributed (i.i.d.) data is violated. In real-world continual learning, data arrives sequentially from shifting distributions (e.g., learning Task A, then Task B). The model's optimization on the new distribution overwrites the parameters tuned for the old one, as there is no mechanism to enforce retention of past knowledge without explicit rehearsal or regularization.

At its core, forgetting is caused by destructive interference in shared model parameters. When gradients for a new task are applied via backpropagation, they indiscriminately update weights that were critical for solving previous tasks. This overwrites the encoded knowledge, as a single set of parameters is forced to represent multiple, potentially conflicting, functions. The degree of forgetting is directly related to the overlap in the parameter subspaces used by different tasks.

Unlike gradual performance decay from concepts like model drift, catastrophic forgetting is characterized by a sharp, dramatic drop in accuracy on previous tasks. This collapse can happen after just a few training iterations or batches on the new data. The model's performance on the original task can plummet to near-random levels, effectively erasing its prior capability. This makes it a critical reliability risk for production systems that must learn continuously.

Catastrophic forgetting is the extreme manifestation of failing to balance the stability-plasticity dilemma. Stability refers to a network's ability to retain existing knowledge. Plasticity is its capacity to learn new information. Standard neural networks are highly plastic but lack stability. Continual learning algorithms aim to introduce stability through techniques like:

• Regularization (e.g., EWC, SI) to penalize changes to important weights.
• Replay of past data to approximate the i.i.d. assumption.
• Architectural isolation to dedicate parameters to specific tasks.

Forgetting is intimately linked to the concepts of transfer learning in a sequential setting.

• Forward Transfer (FWT): Positive influence of learning Task A on the performance of future Task B. This is often the goal of pre-training.
• Backward Transfer (BWT): Influence of learning Task B on the performance of previously learned Task A. Catastrophic forgetting is defined by strongly negative BWT. A key objective of mitigation techniques is to minimize negative BWT and, ideally, achieve positive BWT, where new learning actually consolidates or improves old knowledge.

The severity of forgetting is not constant; it depends heavily on the learning scenario and task relationships.

• Task Similarity: Forgetting is less severe when new tasks are semantically or structurally similar to old tasks (positive forward transfer). It is most severe for dissimilar or conflicting tasks.
• Learning Protocol: Forgetting is more pronounced in Class-Incremental Learning (new classes arrive) and Domain-Incremental Learning (input distribution shifts) than in Task-Incremental Learning, where a task ID is provided at inference.
• Data Order: The sequence of tasks significantly impacts final performance, a phenomenon known as catastrophic interference.

Continual learning is the overarching machine learning paradigm where a model learns sequentially from a non-stationary stream of data or tasks. The primary objective is to accumulate knowledge over time, integrating new information while preserving performance on previously encountered data. It is the field dedicated to solving catastrophic forgetting. Key scenarios include:

• Task-Incremental Learning: Learning distinct tasks with clear boundaries.
• Class-Incremental Learning: Learning new classes over time.
• Domain-Incremental Learning: Adapting to changing data distributions.

The stability-plasticity dilemma is the fundamental trade-off at the heart of catastrophic forgetting and continual learning. Stability refers to a system's ability to retain previously learned knowledge and resist disruptive interference. Plasticity is its capacity to integrate new information and adapt flexibly. Neural networks are inherently plastic during training, which causes instability (forgetting). All continual learning algorithms are, in essence, mechanisms for optimally balancing this dilemma to achieve stable yet plastic learning.

This family of algorithms mitigates catastrophic forgetting by adding a penalty term to the loss function during new task training. This penalty constrains the movement of model parameters deemed important for previous tasks. The core idea is to identify and protect critical weights.

Key Algorithms:

• Elastic Weight Consolidation (EWC): Uses the diagonal of the Fisher Information Matrix to estimate parameter importance.
• Synaptic Intelligence (SI): Computes an online importance measure for each synapse based on the cumulative weight change throughout training.
• Learning without Forgetting (LwF): Uses knowledge distillation with the model's own outputs as soft targets, acting as a regularization signal.

These methods directly address the breakdown of the independent and identically distributed (i.i.d.) data assumption by interleaving past experiences with new data. They store or generate samples from previous tasks in a memory buffer for rehearsal.

Key Techniques:

• Experience Replay (ER): Stores a subset of raw input-output pairs in a fixed-size buffer for replay.
• Generative Replay: Uses a trained generative model (e.g., GAN, VAE) to produce synthetic data from past distributions.
• Dark Experience Replay (DER): Stores the model's logit outputs alongside inputs, enforcing consistency on these "dark knowledge" signals.

These strategies prevent catastrophic forgetting by dynamically modifying the neural network's structure to allocate dedicated, non-overlapping capacity for new tasks. This approach enforces parameter isolation to avoid interference completely.

Key Architectures:

• Progressive Neural Networks: Instantiate new network "columns" for each task, with lateral connections to previous frozen columns.
• Hard Attention to the Task (HAT): Learns binary attention masks over network units, gating activation and gradient flow per task.
• Adapter Layers / Sparse Networks: Add small, task-specific parameters while keeping the core model frozen or sparsely activating pathways.

Backward Transfer is a crucial quantitative metric for evaluating continual learning algorithms and the severity of catastrophic forgetting. It measures the influence of learning a new task on the performance of all previously learned tasks.

Interpretation:

• Negative BWT: Indicates catastrophic forgetting. Performance on old tasks has degraded due to learning the new task.
• Zero BWT: Indicates perfect stability—old task performance is preserved, but no synergy.
• Positive BWT: The ideal outcome, indicating positive knowledge consolidation. Learning the new task has actually improved performance on past tasks.