Experience Replay (ER)

The replay buffer is a fixed-size memory store that retains a subset of past training examples. Its management is critical and involves:

• Sampling Strategy: Common methods include uniform random sampling or prioritized sampling based on metrics like loss or temporal difference error.
• Buffer Update Policy: Deciding which new experiences to store (e.g., reservoir sampling for infinite streams, FIFO replacement).
• Buffer Size: A key hyperparameter balancing retention efficacy against memory and computational overhead. Even a small buffer (e.g., 1% of total data) can yield significant forgetting reduction.

ER's core training loop alternates between learning from new data (plasticity) and rehearsing stored data (stability). In a single training step or mini-batch, the model's gradient update is computed on a mixture of new and replayed samples. This interleaving directly combats the stability-plasticity dilemma by:

• Providing a regularizing signal that anchors parameters to previously learned representations.
• Preventing drastic parameter shifts that would overwrite old knowledge.
• Effectively performing multi-task learning on the current task and a compressed proxy of past tasks.

In online continual learning, data arrives in a temporally correlated stream (e.g., sequential tasks). Standard sequential training violates the independent and identically distributed (IID) data assumption of stochastic gradient descent. ER addresses this by:

• Breaking Correlations: Random sampling from the buffer decorrelates sequential experiences, leading to more stable and efficient gradient descent.
• Smoothing Learning: It prevents the model from overfitting to the most recent data distribution, which is a primary driver of catastrophic forgetting.
• This mechanism is directly inspired by its successful use in Deep Q-Networks for reinforcement learning, where it stabilizes training by breaking correlations in the state-action-reward sequence.

ER introduces explicit trade-offs that must be engineered:

• Memory Overhead: Requires storage for raw data (images, text) or feature representations. This can be prohibitive for high-dimensional data, leading to variants like storing compressed features or latent representations.
• Computational Cost: Each training iteration involves forward/backward passes on replayed data, increasing wall-clock time per epoch.
• Data Privacy: Storing raw past data may be infeasible in privacy-sensitive applications (e.g., healthcare), prompting the use of generative replay or federated continual learning as alternatives.
• The trade-off is typically favorable, as the cost of ER is often far lower than the cost of full retraining on all past data.

ER is loosely inspired by theories of memory consolidation in neuroscience, specifically hippocampal replay observed in rodents. During rest or sleep, the brain reactivates sequences of neural activity corresponding to recent experiences. This hippocampal-neocortical dialogue is believed to transfer and consolidate memories, preventing interference. While the machine learning implementation is simpler, the core principle—reactivating past patterns to stabilize long-term learning—is analogous. This provides a functional justification for the technique beyond its empirical success.

Basic uniform ER has been extended for improved efficiency and performance:

• Dark Experience Replay (DER): Stores the model's logit outputs alongside input-label pairs, using them for a stronger consistency-based distillation loss during replay.
• Gradient Episodic Memory (GEM): Uses the replay buffer to define constraints; the new task's gradient is projected to avoid increasing the loss on past examples.
• Maximally Interfered Retrieval: Actively samples from the buffer the examples most likely to be forgotten (experiencing the largest increase in loss) given the current model parameters.
• Generative Replay: Uses a generative model (e.g., a GAN or VAE) trained on past data to produce synthetic samples for rehearsal, eliminating the need to store raw data.

Experience Replay was popularized by Deep Q-Networks (DQN) to stabilize training. It breaks the temporal correlation of sequential observations by randomly sampling from a replay buffer. This provides several key benefits:

• Decouples learning from experience collection, enabling off-policy algorithms.
• Increases data efficiency by reusing each experience multiple times.
• Smooths training and reduces variance by averaging over many past states. It is a standard component in algorithms like DDPG, TD3, and SAC.

In supervised continual learning, ER is used to rehearse examples from old classes while learning new ones. A small, fixed-size exemplar memory stores a representative subset of past data. During training on a new task:

• The model interleaves mini-batches of new data with replayed old data.
• This rehearsal directly combats catastrophic forgetting by providing a direct signal for previous tasks.
• It is often combined with knowledge distillation (e.g., in iCaRL) where the model's past predictions on replayed data serve as soft targets.

In strict online continual learning, where data is seen only once, a replay buffer acts as a short-term memory. It enables:

• Multi-epoch learning on a static memory subset, improving convergence on complex patterns.
• Balanced training on imbalanced data streams by oversampling rare classes from memory.
• Mitigation of temporal bias from non-stationary data distributions. Algorithms like Gradient Episodic Memory (GEM) use the buffer to compute constraints, ensuring new updates do not increase loss on past experiences.

When storing raw data is infeasible (e.g., for privacy or memory), a generative model (like a Variational Autoencoder or Generative Adversarial Network) can be trained on past data. This model then synthesizes pseudo-examples for rehearsal.

• The core system uses a dual-model architecture: a task model and a generative memory model.
• As new tasks arrive, the generative model is updated to learn the new data distribution.
• This approach provides a more compact, albeit approximate, memory but introduces the challenge of mode collapse in the generator.

ER helps models adapt to new domains without forgetting old ones. By maintaining a buffer with samples from multiple previous domains, the model rehearses a mixture of distributions.

• This practice improves backward transfer (performance on old domains) and can enhance forward transfer to new, related domains.
• It is particularly useful in federated continual learning, where devices with different data distributions use local replay buffers to maintain performance on global tasks while adapting to local data.

Experience Replay serves as a strong baseline in continual learning research. Its simplicity and effectiveness make it a critical point of comparison for more complex methods.

• Standard benchmarks like Split CIFAR-100 and Streaming ImageNet report ER performance.
• Research often measures the trade-off between buffer size and final accuracy.
• Advanced variants like Dark Experience Replay (DER), which stores and replays logits, demonstrate how the core idea can be extended for stronger regularization.