Dreamer

Dreamer's core is a Recurrent State-Space Model (RSSM), a latent dynamics model that learns a compact, abstract representation of the environment. It encodes high-dimensional observations (like images) into a stochastic latent state combined with a deterministic recurrent state. This model predicts future latent states and rewards, enabling long-horizon imagination in a computationally efficient, compressed space. This architecture is crucial for generalizing from pixels and managing partial observability.

Unlike algorithms that plan online, Dreamer trains its policy and value function entirely from imagined rollouts. Starting from encoded real-world states, it uses its RSSM to simulate hundreds of steps into the future. Backpropagation Through Time (BPTT) is applied through these latent trajectories to compute gradients for the actor and critic networks. This decouples policy training from real environment interaction, making learning exceptionally sample-efficient as a single real experience can seed countless informative imaginations.

Dreamer optimizes its world model not just for accurate next-state prediction, but for value-aware prediction. The model loss includes terms for reconstructing observations, predicting rewards, and predicting task continuation. Crucially, it also learns to predict the value of latent states. This shapes the latent space to be informative for control, ensuring that imagined rollouts are relevant for policy optimization, not just physically accurate. This is a key distinction from pure system identification.

The RSSM's explicit stochastic latent variable allows Dreamer to model aleatoric uncertainty inherent in real environments. By sampling from this distribution during imagination, the agent considers multiple plausible futures. This prevents the policy from overfitting to a single, deterministic prediction and leads to more robust behaviors that can handle randomness and partial observability. The stochastic pathway is regularized with a KL divergence term to maintain a manageable latent space.

A critical hyperparameter is the imagination horizon (H), the number of steps simulated for each training iteration. A longer horizon allows the policy to optimize for long-term rewards but increases computational cost and the risk of compounding model error. Dreamer typically uses horizons of 15-50 steps, finding a balance where the model is accurate enough for useful long-term gradients. This is a fundamental engineering trade-off between foresight and fidelity in model-based RL.

• vs. MBPO (Model-Based Policy Optimization): MBPO uses short model rollouts to generate synthetic data for a model-free RL algorithm (like SAC). Dreamer, in contrast, directly backpropagates through the model to train the policy, offering a more integrated approach.
• vs. MuZero: MuZero learns a value-equivalent model focused on predicting policy, value, and reward for planning. Dreamer learns a latent dynamics model that also reconstructs observations and is used for direct gradient-based policy training, not Monte Carlo Tree Search.

A world model is an agent's internal, learned representation that predicts future environment states and rewards. It acts as a compressed, abstract simulator, enabling planning and imagination without direct, costly interaction with the real world. In Dreamer, this is implemented as a Recurrent State-Space Model (RSSM).

• Core Function: Encodes high-dimensional observations (e.g., pixels) into a latent state and predicts future latent states and rewards.
• Key Benefit: Allows the agent to "dream" or conduct imagined rollouts to train its policy efficiently.

The Recurrent State-Space Model (RSSM) is the specific latent dynamics model architecture at the heart of Dreamer. It combines deterministic recurrence with stochastic latent variables to model temporal dependencies in partially observable environments.

• Architecture: Uses a deterministic recurrent network (like a GRU) to track history and a stochastic latent variable to represent uncertainty about the current state.
• Purpose: Projects high-dimensional observations (images) into a compact latent space where dynamics are learned and imagined rollouts are computationally feasible.

A latent dynamics model learns to predict future states in a compressed, abstract latent space rather than the raw, high-dimensional observation space (e.g., pixel space). This is a cornerstone of Dreamer's sample efficiency.

• Advantage over Pixel Models: Dramatically reduces computational complexity and improves generalization by learning essential features.
• Process: The encoder compresses an image into a latent vector; the dynamics model predicts the next latent vector given an action.
• Use Case: Enables long-horizon imagined rollouts via simple matrix multiplications in latent space.

Imagined rollouts (or latent imagination) are synthetic trajectories of states, actions, and rewards generated by unrolling the learned world model from a starting state. Dreamer trains its actor and critic networks exclusively on these rollouts.

• Mechanism: Starting from a real environment observation encoded into the latent space, the policy proposes actions, and the RSSM predicts the next latent state and reward.
• Training Loop: The policy is improved via backpropagation through time (BPTT) on these imagined sequences to maximize predicted reward.
• Benefit: Eliminates the need for expensive model-free algorithms like PPO to interact with the real environment for policy training.

Model-policy co-adaptation is a critical failure mode in MBRL where a policy overfits to the specific biases and inaccuracies of its own learned dynamics model. This leads to excellent performance in the model's simulation but catastrophic failure in the real environment.

• Cause: The policy exploits shortcuts or errors in the model that do not exist in reality.
• Dreamer's Mitigation: By using a latent model and training the policy via gradient-based optimization on imagined trajectories (rather than aggressive planning), it regularizes the policy to be more robust to small model errors.
• Contrast: Compared to certainty-equivalence control, which blindly trusts the model.

Sample efficiency measures the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the primary claimed advantage of model-based RL algorithms like Dreamer over model-free methods.

• Metric: Often measured in environment steps or episodes needed to reach a performance threshold.
• Dreamer's Approach: Achieves high sample efficiency by learning a compact world model from limited real data, then using it to generate a vast amount of synthetic training data (imagined rollouts) for the policy.
• Result: Can learn complex behaviors from orders of magnitude fewer real environment interactions than model-free counterparts like DQN or PPO.