Experience Replay

The Replay Buffer is the fixed-capacity memory storage at the heart of the system. It stores past experiences, typically as tuples of (state, action, reward, next state, done flag). Its primary engineering challenge is buffer management: deciding which experiences to store, retain, or discard as new data streams in. Common strategies include FIFO (First-In, First-Out) for a simple sliding window of recent history, or more sophisticated methods like reservoir sampling to maintain a statistically uniform sample from the entire data stream seen so far. The buffer's size is a critical hyperparameter, trading off rehearsal diversity against memory constraints, especially on edge devices.

The Sampling Strategy defines the algorithm for selecting experiences from the buffer for rehearsal. Uniform random sampling is standard, providing unbiased rehearsal. However, prioritized sampling is a powerful variant where experiences are weighted by their Temporal-Difference (TD) error—a measure of how 'surprising' or informative the experience was. This focuses rehearsal on experiences the model hasn't yet mastered, improving sample efficiency. The strategy must be computationally lightweight to avoid becoming a bottleneck during on-device training cycles.

This component governs the training regimen. Instead of training exclusively on new task data, the Interleaving & Rehearsal Loop mixes mini-batches. A typical update might consist of 50% new data and 50% old data sampled from the replay buffer. This interleaving forces the model's optimization process to satisfy two objectives simultaneously:

• Plasticity: Adjust parameters to minimize loss on the new task.
• Stability: Preserve parameters to minimize loss on rehearsed old tasks. This direct rehearsal is the primary mechanism for mitigating catastrophic forgetting by continuously reminding the model of prior knowledge.

Closely tied to buffer management, the Buffer Update Policy is the rule set for modifying the buffer's contents when new experiences arrive. For a fixed-size buffer, each new experience must trigger a decision: ignore it, add it (and potentially eject an old one), or replace an existing entry. Policies range from simple (e.g., always add, eject the oldest) to complex, goal-oriented strategies. Examples include:

• Core-set Selection: Adding experiences that are representative prototypes of the data distribution.
• Uncertainty-Based: Retaining samples where the model's prediction confidence is low.
• Gradient-Based: Keeping samples that induced large parameter updates. The policy is crucial for maintaining a high-utility, non-redundant memory under strict storage limits.

A key design choice is what to store in the buffer. Raw Data Storage retains the original input (e.g., an image pixel tensor), which is simple but memory-intensive. Representation Storage is an advanced, memory-efficient alternative where only the latent feature vector from an intermediate layer of the network is stored, alongside the label/target. During rehearsal, this vector is fed into the later layers of the network. This can reduce storage footprint by orders of magnitude—critical for edge deployment—but requires a stable feature extractor; if early layers change drastically, old representations may become invalid.

Experience Replay's effectiveness is realized through the Loss Function. The total loss for a training step is typically a weighted sum: L_total = L_new_task + λ * L_replay Where L_replay is the loss computed on the batch sampled from the buffer. This formalizes the stability-plasticity trade-off. The hyperparameter λ controls the strength of rehearsal. In advanced schemes, L_replay may not be a standard cross-entropy; it could be a knowledge distillation loss, comparing current outputs to the outputs generated by a frozen snapshot of the old model, further stabilizing the learned representations.

The foundational strategy where experiences are sampled uniformly at random from the buffer. This assumes all past data is equally valuable for rehearsal.

• Mechanism: When the buffer is full, new experiences typically overwrite old ones at random (FIFO) or via reservoir sampling to maintain a statistically uniform random sample of the entire stream.
• Advantage: Simple, unbiased, and provides a baseline. It helps decorrelate sequential data and smooths training.
• Limitation: Does not prioritize informative or challenging examples, which can be inefficient for learning.

Samples experiences with probability proportional to their temporal-difference (TD) error, a measure of how 'surprising' or incorrect the model's prediction was.

• Mechanism: Each experience (s, a, r, s') is stored with its last computed TD error. Sampling uses a stochastic prioritization formula: P(i) = p_i^α / Σ_k p_k^α, where p_i = |δ_i| + ε. The hyperparameter α controls the prioritization strength.
• Importance Sampling: To correct the bias introduced by non-uniform sampling, updates are weighted by the inverse probability of being sampled.
• Impact: Focuses rehearsal on poorly understood experiences, often leading to faster convergence in reinforcement learning contexts.

A probabilistic algorithm designed to maintain a uniform random sample of fixed size k from a data stream of unknown or infinite length.

• Algorithm: For the i-th element in the stream (where i starts at 1):
  1. If i ≤ k, insert the element into the buffer.
  2. If i > k, with probability k / i, replace a randomly chosen element in the buffer with the new element.
• Property: After processing n elements, every element has an exactly k / n probability of being in the buffer, ensuring a statistically fair sample of the entire history.
• Use Case: The standard buffer management strategy for online continual learning where data is seen only once.

Aims to store a minimal subset of past data that best represents the entire dataset for a given task, optimizing for coverage and diversity.

• Objective: Solve a k-center or k-medoids problem to select k exemplars that minimize the maximum distance between any data point and its nearest exemplar in the feature space.
• Process: Often uses the model's penultimate layer embeddings to compute distances in a semantically meaningful space.
• Advantage: Maximizes the representational power of a fixed-size buffer, making rehearsal more efficient. It is a key strategy in class-incremental learning algorithms like iCaRL.
• Challenge: Computationally expensive; often requires periodic re-computation of the core-set.

Selects experiences for the buffer based on their estimated contribution to the loss gradient or their influence on model parameters.

• Gradient of Loss: Prefers examples that produce large gradient magnitudes, as they are likely to induce significant learning.
• Influence Functions: Estimates how much the model's predictions would change if a specific training example were removed. Examples with high influence are candidates for retention.
• Gaussian Mixture Modeling: Models the distribution of gradients and samples experiences to match the full gradient distribution, ensuring the rehearsal batch provides a faithful gradient signal.
• Application: Used in advanced rehearsal methods to combat catastrophic forgetting by ensuring the buffer's gradient aligns with that of the full past dataset.

Ensures the replay buffer maintains a balanced representation across all learned tasks, which is critical for class-incremental and task-incremental learning scenarios.

• Strategy: The buffer is partitioned per task. During rehearsal, samples are drawn evenly from each task's partition, or proportionally based on task difficulty/age.
• Herding: A technique used to select representative exemplars for each class that approximate the class mean in feature space, as seen in iCaRL.
• Challenge: Requires task identity at training time (relaxed in some advanced methods) and careful management as the number of tasks grows.
• Goal: Prevents the model from biasing its predictions toward recently learned tasks, promoting stability across the entire task sequence.