Model-Based Experience Replay

The core component is a world model—a neural network trained to predict future states and rewards in a compressed latent space. This model acts as a synthetic data generator, creating plausible experience tuples (state, action, reward, next state) without interacting with the real environment. This decouples data collection from policy learning, dramatically increasing sample efficiency by reusing model-generated data for thousands of gradient updates.

Agents like Dreamer train their policies entirely within the model's latent imagination. Instead of rolling out trajectories in the high-dimensional environment (e.g., pixels), the agent plans in the compact latent space of the world model. These imagined trajectories are then replayed to train the actor and critic networks. This process, known as latent planning, reduces computational cost and enables the agent to evaluate millions of potential futures quickly.

A critical challenge is model bias, where inaccuracies in the world model are exploited by the policy, leading to catastrophic failure in the real environment. Model-based experience replay addresses this through:

• Regularized updates that penalize policies for exploiting model flaws.
• Ensemble methods using multiple world models to quantify uncertainty.
• Mixed-buffer sampling, where real environment transitions are interleaved with synthetic ones to ground learning in reality.

The world model enables directed exploration. By quantifying its own prediction uncertainty (e.g., via ensemble disagreement), the agent can prioritize exploring state-action pairs where the model is least confident. These high-uncertainty, informative synthetic experiences are added to the replay buffer, guiding the policy to explore novel regions of the state space more efficiently than random, undirected exploration.

Model-based experience replay is often hybridized with model-free algorithms. For example, MuZero learns a model implicitly for planning but stores and samples real trajectories in its replay buffer to train its representation, dynamics, and prediction networks. This combines the strategic foresight of model-based planning with the robust policy optimization of model-free methods like policy gradients or Q-learning.

Beyond reinforcement learning, this paradigm is applied in continual learning to combat catastrophic forgetting. Here, a generative world model is trained on data from previous tasks. When learning a new task, the model replays synthetic data from old tasks, interleaving it with new real data. This approximates the i.i.d. data assumption of batch learning, allowing the network to retain performance on past tasks without storing raw data.

A world model is a learned neural network that predicts future environment states and rewards from a compressed latent representation. It is the core generative component in model-based experience replay, used to synthesize plausible future trajectories for the agent to learn from, drastically improving sample efficiency.

• Function: Encodes the environment's dynamics without explicit rules.
• Output: Predicts next latent state, reward, and episode continuation signal.
• Key Benefit: Enables planning and imagination within a learned, abstract simulation.

An experience replay buffer is a fixed-size data structure (often a circular buffer) that stores past agent experiences as tuples (state, action, reward, next state). It decouples the data generation process from learning by allowing random sampling of past transitions, which breaks harmful temporal correlations and stabilizes training.

• Core Purpose: Provides a diverse, de-correlated dataset for off-policy learning.
• In Model-Based Replay: Stores both real interactions and synthetic trajectories generated by the world model.
• Implementation: Typically uses FIFO (First-In-First-Out) logic when full.

Generative replay is a continual learning technique where a separate generative model (e.g., a Variational Autoencoder or GAN) is trained to produce synthetic data samples from previous tasks. These generated samples are interleaved with new task data during training to mitigate catastrophic forgetting.

• Relation to Model-Based Replay: Both use generative models to create training data. Generative replay is task-agnostic and used for supervised/continual learning, while model-based replay is for RL and uses a dynamics model.
• Key Distinction: The world model in MBRL predicts sequential dynamics, not just static samples.

Dreamer is a seminal model-based reinforcement learning agent that learns a world model from pixels and uses it to train an actor-critic policy entirely within its generated latent imagination. It is a prime example of model-based experience replay in action.

• Mechanism: The agent "dreams" (rolls out) trajectories in the latent space of its world model.
• Replay: These dreamed trajectories are stored in a replay buffer and used to compute policy and value gradients.
• Result: Achieves high sample efficiency by learning from imagined experience.

Off-policy correction refers to algorithms that adjust learning targets to account for the difference between the behavior policy that collected the data and the target policy being improved. This is critical when using a replay buffer, as stored experiences are generated by older policies.

• Examples: Retrace(λ), V-trace, Importance Sampling.
• Role in Model-Based Replay: When a world model generates synthetic trajectories using the current policy, it creates on-policy data, reducing but not eliminating the need for off-policy corrections on older real data in the buffer.

Catastrophic forgetting (or catastrophic interference) is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. Experience replay is a primary defense mechanism against this phenomenon in continual learning.

• Challenge: Neural weights optimized for Task B overwrite those crucial for Task A.
• How Replay Helps: By interleaving old experiences (real or synthetic) with new data, the network is continually reminded of past tasks.
• Model-Based Angle: A world model can generate unlimited "reminders" of past environment dynamics, providing a powerful tool to combat forgetting in non-stationary RL.