Catastrophic Forgetting

Continual learning, or lifelong learning, is a machine learning paradigm where a model learns sequentially from a stream of tasks or data distributions without forgetting previously acquired knowledge. It is the primary research field addressing catastrophic forgetting.

• Key techniques include elastic weight consolidation, gradient episodic memory, and experience replay.
• The goal is to develop plasticity-stability trade-off, allowing models to integrate new information while preserving old.
• It is essential for real-world AI systems that must adapt to changing environments, such as autonomous vehicles or personalized assistants.

Elastic Weight Consolidation is a regularization-based algorithm designed to mitigate catastrophic forgetting in neural networks. It protects important parameters for previous tasks by penalizing changes proportional to their estimated Fisher information.

• The method calculates an importance weight for each network parameter based on its contribution to past task performance.
• During training on a new task, changes to important weights are heavily penalized, effectively "consolidating" old knowledge.
• EWC enables a single network to learn multiple tasks sequentially, making it a foundational technique for continual learning.

Multi-task learning is an approach where a single model is trained simultaneously on multiple related tasks, sharing representations to improve generalization and data efficiency. It is often contrasted with sequential continual learning.

• By learning tasks in parallel, the model inherently avoids catastrophic forgetting as all tasks are present during the full training cycle.
• This requires access to all task data upfront, which is often impractical for streaming or evolving data scenarios.
• It demonstrates the benefits of shared representations but does not solve the core sequential learning problem.

The stability-plasticity dilemma is a fundamental challenge in both biological and artificial learning systems, describing the tension between retaining old knowledge (stability) and integrating new information (plasticity).

• Catastrophic forgetting is the extreme failure mode where plasticity overwhelms stability.
• All continual learning algorithms aim to find an optimal balance, often through explicit regularization, dynamic architectures, or rehearsal of old data.
• This dilemma is central to designing AI systems capable of lifelong adaptation.

Experience replay is a technique to combat catastrophic forgetting by storing a subset of past training data (or generated samples) in a replay buffer and interleaving it with new data during training.

• This method effectively simulates joint training on old and new tasks, preventing the model's parameters from drifting away from previous solutions.
• It is inspired by hippocampal replay in neuroscience and is widely used in both continual learning and deep reinforcement learning.
• The core challenge is managing buffer size and selecting which experiences to retain for maximal knowledge preservation.

Reward overoptimization is a failure mode in reinforcement learning from human or AI feedback (RLHF/RLAIF) where an agent, by maximizing an imperfect proxy reward function too aggressively, leads to a sharp decline in true performance.

• While distinct from classic catastrophic forgetting, it shares the theme of performance collapse due to sequential optimization.
• It often occurs due to distributional shift or reward hacking, where the policy exploits flaws in the reward model.
• Mitigation strategies include KL divergence penalties, reward normalization, and ensemble rewards, which parallel techniques used to stabilize continual learning.