Replay Buffer

A replay buffer stores agent interactions as transition tuples, typically represented as (s, a, r, s'):

• s: The state observed by the agent.
• a: The action taken in that state.
• r: The scalar reward received after the action.
• s': The resulting next state. This uniform structure allows algorithms to treat every interaction as a discrete, self-contained learning example, independent of when it was originally collected.

Sequential data from an agent's trajectory is highly correlated, which violates the independent and identically distributed (i.i.d.) assumption of stochastic gradient descent. By storing experiences and then sampling them uniformly at random, the replay buffer shuffles the data, breaking harmful temporal correlations. This prevents the agent from overfitting to recent, narrow experiences and makes the learning process significantly more stable.

In online reinforcement learning, an experience is typically used once for a model update and then discarded. A replay buffer enables sample reuse, allowing each collected transition to be sampled and learned from many times. This is critical in environments where data collection is expensive or slow, dramatically improving the sample efficiency of algorithms like Deep Q-Networks (DQN).

A standard uniform sampling strategy treats all experiences as equally important. Prioritized Experience Replay (PER) improves upon this by assigning a priority to each transition based on its Temporal Difference (TD) error—a measure of how surprising or unexpected the outcome was. Experiences with higher TD errors are sampled more frequently, allowing the agent to focus its learning on the most informative events and often leading to faster convergence.

The replay buffer is the key mechanism that enables off-policy learning, where an agent learns about an optimal target policy using data generated by a different, often more exploratory, behavior policy. By storing and mixing experiences from various past policies, the agent can learn a superior policy without needing to execute it directly in the environment, a concept fundamental to algorithms like Q-Learning and its deep learning variants.

Replay buffers are typically implemented as circular buffers with a fixed maximum capacity. When full, the oldest experiences are overwritten by new ones. This finite capacity serves two purposes:

• Memory constraint: Prevents unbounded RAM usage on hardware.
• Recency bias: Ensures the agent gradually forgets obsolete experiences from an outdated policy, maintaining a relevant data distribution for continual learning in non-stationary environments.

The seminal algorithm that popularized the replay buffer. DQN combines Q-learning with deep neural networks and uses experience replay to store and randomly sample past transitions. This breaks the temporal correlations in sequential data, satisfying the i.i.d. assumption of stochastic gradient descent. The buffer also enables the use of a separate target network, which is periodically updated to stabilize the learning target.

An enhancement to the standard uniform sampling replay buffer. Instead of treating all transitions equally, this method assigns a priority to each experience based on its temporal-difference (TD) error—the magnitude of the agent's surprise. Transitions with higher TD errors are sampled more frequently, focusing learning on the most informative experiences. This addresses the inefficiency of uniform replay, where many transitions may be redundant or trivial.

The tendency of neural networks to abruptly forget previously learned knowledge when trained on new data distributions. In reinforcement learning, the replay buffer mitigates this by interleaving old experiences with new ones during training. By continuously rehearsing past transitions, the agent maintains performance on earlier states while adapting to new ones. This is a critical function for continual learning agents operating in non-stationary environments.