Experience Replay Buffer

The primary function of an experience replay buffer is to break the temporal correlation between consecutive experiences sampled from the environment. In online learning, sequential states are highly correlated, which can cause the neural network's gradient updates to become unstable and diverge. By storing experiences and randomly sampling mini-batches from across different time steps, the buffer decorrelates the data, approximating the independent and identically distributed (i.i.d.) assumption required for stable stochastic gradient descent. This is fundamental to algorithms like Deep Q-Network (DQN).

Experience replay dramatically increases sample efficiency by allowing each real interaction with the environment to be used for multiple gradient updates. Instead of being discarded after a single update, an experience tuple (s, a, r, s') is stored and can be replayed numerous times. This reuse of data is critical in domains where collecting experience is expensive or time-consuming, such as robotics or real-world simulations. The replay ratio—the average number of times an experience is used—is a key hyperparameter balancing new learning with data reuse.

The buffer is typically implemented as a fixed-capacity circular buffer (or ring buffer). When the buffer reaches its maximum capacity (e.g., 1 million transitions), the oldest experience is overwritten by the newest. This FIFO (First-In-First-Out) structure provides a simple, memory-efficient window into the agent's recent history. For non-stationary environments, more sophisticated sampling strategies like reservoir sampling can be used to maintain a statistically uniform sample of the entire data stream. The core operation is batch sampling, where a random mini-batch is drawn for training.

The replay buffer is the enabling mechanism for off-policy learning algorithms. It allows an agent to learn a target policy (e.g., the greedy policy) from experiences generated by an older or exploratory behavior policy (e.g., an epsilon-greedy policy). This separation of data collection from policy improvement is powerful. However, it introduces bias because the data distribution in the buffer differs from the current policy's distribution. Techniques like importance sampling are used in some algorithms (e.g., SAC, DDPG) to correct this bias when computing gradient updates.

The basic replay buffer is extended by numerous advanced algorithms that modify storage or sampling logic:

• Prioritized Experience Replay (PER): Samples transitions with probability proportional to their Temporal Difference (TD) error, focusing learning on 'surprising' experiences.
• Hindsight Experience Replay (HER): For goal-conditioned RL, it replays failed episodes with the achieved goal as the target, learning from failure.
• Trajectory Buffer: Stores complete sequences for algorithms requiring temporal context (e.g., on-policy RL, Decision Transformer).
• Demonstration Buffer: Stores expert trajectories for imitation or pre-training.

In the context of continual learning, experience replay acts as a dynamic episodic memory. By storing a subset of past experiences and interleaving them with new data during training, the neural network is repeatedly exposed to old tasks, which helps mitigate catastrophic forgetting. This principle is central to algorithms like Gradient Episodic Memory (GEM) and Averaged-GEM (A-GEM), which use the buffer to compute constraints on new learning. A related technique, Generative Replay, uses a generative model to produce synthetic past data for the same purpose.

A sampling strategy where experiences are selected from the replay buffer with a probability proportional to their temporal-difference (TD) error. This focuses learning on transitions where the agent's predictions were most inaccurate, leading to faster convergence. The implementation requires a sum-tree data structure for efficient sampling.

• Key Mechanism: Samples high-error transitions more frequently.
• Trade-off: Introduces bias that must be corrected with importance sampling weights.

A technique for goal-conditioned reinforcement learning that enables learning from failure. When an episode fails to achieve its original goal, HER replays it with the actually achieved final state substituted as a virtual goal. This teaches the agent that its actions are useful for some goals, even if not the intended one.

• Primary Use: Sparse and binary reward environments (e.g., robotic manipulation).
• Core Insight: Treats every final state as a potential goal, dramatically improving sample efficiency.

A replay buffer that stores and samples complete sequences (trajectories) of states and actions, rather than individual, decorrelated transitions. This is essential for training on-policy algorithms (like PPO) or models that require temporal context, such as Recurrent Neural Networks (RNNs) or Decision Transformers.

• Contrast with Standard Buffer: Maintains sequential dependencies.
• Storage: Typically holds (s_0, a_0, r_0, ..., s_T) sequences.

A continual learning technique where a generative model (e.g., a Variational Autoencoder or Generative Adversarial Network) is trained to produce synthetic samples from previous tasks. These generated samples are interleaved with new task data during training to mitigate catastrophic forgetting.

• Application: Class-incremental learning scenarios.
• Function: Simulates a 'replay buffer' for tasks where raw data cannot be stored due to privacy or storage constraints.

Uses a learned world model—a neural network that predicts the next state and reward—to generate synthetic experiences. These imagined trajectories are added to the replay buffer, augmenting real data. This hybrid approach improves sample efficiency and aids in planning.

• Examples: The Dreamer and MuZero algorithms.
• Benefit: Allows the agent to 'practice' in its own internal simulation.

The seminal algorithm that popularized experience replay for deep reinforcement learning. DQN combined Q-Learning with deep neural networks and introduced two key stabilizers: the experience replay buffer and a target network. The buffer breaks temporal correlations, while the target network provides stable Q-value targets.

• Historical Impact: First deep RL agent to master multiple Atari games from pixels.
• Core Architecture: The foundation for advanced variants like Rainbow DQN.