Experience Replay

The replay buffer (or memory buffer) is the core data structure. It is a finite, first-in-first-out (FIFO) queue that stores transition tuples, typically in the form (state, action, reward, next state, done flag). This buffer serves two primary functions:

• Breaking Temporal Correlations: By randomly sampling past experiences, it removes the sequential dependency present in online learning, stabilizing gradient updates.
• Data Reuse: High-value or rare experiences can be learned from multiple times, dramatically improving sample efficiency compared to purely online methods like standard Q-learning.

Sampling strategy is a key design choice:

• Uniform Sampling: The original method. Transitions are selected from the buffer with equal probability. Simple and unbiased but may be inefficient.
• Prioritized Experience Replay (PER): A critical enhancement. Transitions are sampled with probability proportional to their temporal-difference (TD) error—the magnitude of the prediction surprise. This focuses learning on 'surprising' or poorly predicted experiences. It requires importance-sampling weights to correct for the bias introduced by non-uniform sampling.

Experience replay was instrumental in the success of Deep Q-Networks. DQN combined it with a target network to address non-stationarity. The algorithm uses two neural networks:

• The online network (Q-network) that is updated every step.
• A target network (Q-target) that provides stable regression targets and is periodically copied from the online network. By sampling mini-batches from the replay buffer and using the target network for the max Q(next_state) calculation, DQN achieves stable learning with deep neural networks, a previously intractable problem.

Experience replay naturally extends to multi-step learning. Instead of storing single-step transitions, you can store sequences or compute n-step returns directly. For example, an (state, action) pair can be stored with the discounted sum of the next n rewards and the state n steps ahead. This blends the low variance of Monte Carlo methods with the bias of one-step Temporal Difference learning, often leading to faster policy improvement and better credit assignment over longer time horizons.

The replay buffer is the engine of off-policy learning. It allows an agent to learn a target policy (e.g., the optimal policy) from experiences generated by a different behavior policy (e.g., an exploratory policy). This separation is crucial for:

• Safe Exploration: Learning from exploratory data generated in simulated or safe environments.
• Parallel Data Collection: Using multiple actors or past data to fill the buffer asynchronously.
• Reusing Demonstrations: Incorporating expert trajectories (via hindsight experience replay or demonstration buffers) to bootstrap learning.

In model-based reinforcement learning, experience replay takes on additional roles:

• Dynamics Model Training: The buffer provides the dataset (state, action, next_state) for supervised learning of the environment's transition function.
• Model-Based Planning: Algorithms like Model-Based Policy Optimization (MBPO) use short rollouts from a learned model, with the resulting synthetic data being stored in a replay buffer for policy training.
• World Model Learning: In architectures like the Dreamer agent, the replay buffer stores sequences of past observations and actions used to train a latent dynamics model (the world model) and the actor-critic components entirely in latent space.

A world model is an internal, learned representation within an AI system that captures the dynamics and regularities of its environment. It enables the agent to simulate and predict future states without direct interaction, forming the predictive target for data sampled via experience replay.

• Core Function: Serves as a compressed, forward-looking simulator.
• Training Data: Often trained on transitions (state, action, next_state) stored in the replay buffer.
• Use Case: In model-based reinforcement learning, the world model is used for planning by 'imagining' rollouts.

Model-based reinforcement learning (MBRL) is an approach where an agent learns an explicit model of the environment's dynamics (the transition and reward functions). Experience replay is a foundational component for MBRL, as it provides the diverse, decorrelated data needed to train an accurate and generalizable world model.

• Key Distinction: Contrasts with model-free RL, which learns a policy or value function directly.
• Role of Replay: The replay buffer acts as the dataset for the world model learner.
• Advantage: Dramatically improves sample efficiency by enabling planning in the learned model.

A POMDP is the mathematical framework for sequential decision-making under uncertainty, where the agent cannot directly observe the true environment state. Experience replay in POMDPs often stores sequences of observations and actions. The agent must learn to infer a latent state or belief state from this history, which is a primary goal of world model learning.

• Core Challenge: The agent only receives partial, noisy observations.
• Replay Content: May store trajectories of observations to learn temporal dependencies.
• Solution: World models in POMDPs learn to map observation histories to predictive latent states.

A latent state is a compressed, often unobservable, representation of an environment's true condition, inferred from raw sensory data (e.g., pixels, sensor readings). World model learning aims to discover informative latent states. Experience replay provides the temporal data needed to learn that successive latent states evolve predictably given an action.

• Purpose: Distills noisy, high-dimensional observations into a succinct representation for reasoning.
• Learning Method: Often discovered via self-supervised learning on replay buffer sequences.
• Connection: In a POMDP, the latent state approximates the ideal, fully-observed state.

Self-supervised learning (SSL) is a paradigm where a model generates its own supervisory signals from the structure of unlabeled data. World models are typically trained using SSL objectives on data from the experience replay buffer. Common objectives include predicting the next latent state, reconstructing the observation, or contrasting similar vs. dissimilar transitions.

• Primary Data Source: The unlabeled experience replay buffer.
• Common Techniques: Next-frame prediction, contrastive learning, and autoencoding.
• Goal: To learn general, reusable representations (latent states) of the environment.

Continual learning is the ability of a model to learn sequentially from a non-stationary stream of data without catastrophic forgetting. Experience replay is a key algorithmic defense against forgetting in continual learning for RL agents. By maintaining and repeatedly sampling from a buffer of past experiences, the agent rehearses old tasks while learning new ones.

• Core Problem: Catastrophic forgetting of old skills when learning new ones.
• Replay's Role: The buffer acts as explicit memory of past tasks/distributions.
• Challenge: Balancing buffer size and content to cover a changing world model.