Generative Replay

The core mechanism of Generative Replay is the use of a generative model—typically a Generative Adversarial Network (GAN) or Variational Autoencoder (VAE)—to produce artificial training examples that mimic the data distribution of previously learned tasks. This synthetic data is interleaved with real data from the current task during training. The generative model itself must be trained continually, often using its own generated outputs from earlier tasks as part of its training data to prevent its own forgetting.

A key advantage of Generative Replay is its model-agnostic nature. The replay mechanism is external to the primary classifier or policy network being trained. This means:

• The main model's architecture does not need to be modified, expanded, or masked.
• There is no need for task-specific parameters or complex routing logic.
• The same generative replay pipeline can be applied to different model architectures (CNNs, Transformers) with minimal changes. This separates the concerns of knowledge retention from the primary model's design.

By continuously supplementing the current task's data stream with generated samples from all past tasks, Generative Replay aims to reconstruct an approximately independent and identically distributed (i.i.d.) dataset. This directly counters the non-i.i.d., sequential nature of continual learning streams. The training objective becomes: L_total = L_new_task(θ) + L_replay(θ) where L_replay is the loss computed on the synthetic data from the generator. This allows standard stochastic gradient descent to be applied effectively.

A major challenge is catastrophic forgetting within the generator itself. If the generative model forgets how to produce data from early tasks, the entire replay mechanism fails. Solutions include:

• Training the generator with its own replayed outputs in a recursive manner.
• Using a separate generator for each task, though this increases memory cost.
• Employing generative models with strong latent disentanglement (e.g., certain VAEs) to improve stability. The quality and diversity of the generated samples are critical to the success of the overall method.

Generative Replay is often contrasted with Exemplar Replay (or Experience Replay), which stores a subset of real past data in a fixed-size buffer.

Several algorithmic variants build upon the core generative replay concept:

• Deep Generative Replay: The seminal work using a deep generative model (a VAE-GAN) to replay data for a classifier.
• Brain-Inspired Replay (BI-R): Uses generative feedback connections in a neural network to produce replay activity.
• Latent Replay: Replays intermediate feature representations (latent vectors) instead of raw input data, which is more efficient and often more stable.
• Conditional Generative Replay: Uses task or class labels to condition the generator, improving the relevance and control of the synthesized data. These variants address specific challenges in scalability, efficiency, and stability.

The foundational replay-based method upon which Generative Replay builds. Experience Replay stores a small, fixed-size buffer of raw data samples from previous tasks. During training on a new task, it interleaves these stored exemplars with new data, approximately maintaining the i.i.d. data assumption and providing direct rehearsal of past distributions. Its primary limitation is the memory buffer size, which constrains the amount of past knowledge that can be retained.

An advanced variant of Experience Replay that stores more than just input-output pairs. Dark Experience Replay saves the model's logit outputs (the 'dark knowledge') for each stored input. During rehearsal, it applies a consistency loss (e.g., MSE) between the current model's logits and the stored logits. This anchors the model's behavior on previous tasks more effectively than a standard classification loss, often leading to better retention with the same memory budget.

A leading regularization-based method that takes a fundamentally different approach from replay. EWC adds a quadratic penalty term to the loss function during new task training. This penalty constrains the movement of parameters deemed important for previous tasks, with importance estimated via the diagonal of the Fisher Information Matrix. It represents the 'stability' side of the dilemma, enforcing rigidity on critical synapses without requiring stored data.

A knowledge distillation-based method that avoids storing past data. When learning a new task, LwF passes the new data through a frozen copy of the model from the previous task step. The outputs (logits) become soft targets for the old tasks. The updated model is then trained with a combined loss for the new task labels and a distillation loss to match the old task soft targets. It is a rehearsal-free method but can struggle with significant distribution shifts.

A replay-based method that uses stored exemplars to directly constrain gradient updates. GEM maintains an episodic memory of past examples. Before applying a gradient step for a new task, it computes the proposed update and checks if it would increase the loss on the memory buffer. If so, it projects the gradient onto the closest direction (solving a quadratic program) that does not increase the past loss. This provides a hard guarantee against negative backward transfer on the stored memories.

The fundamental challenge that all continual learning methods, including Generative Replay, aim to solve. It describes the tension between:

• Stability: The system's ability to retain learned knowledge and resist catastrophic forgetting.
• Plasticity: The system's ability to integrate new information and adapt to novel tasks or data distributions. Generative Replay addresses this by using a generative model's plasticity to create data for stable rehearsal, explicitly balancing both requirements within a unified framework.