Dreamer

The core of Dreamer is its Recurrent State-Space Model (RSSM), a generative world model that learns a temporally coherent latent representation. It comprises:

• Representation Model: Encodes an image observation into a stochastic latent state.
• Transition Model: Predicts the next latent state given the previous state and action.
• Observation Model: Reconstructs the image from the latent state.
• Reward Model: Predicts the expected reward for a given latent state. This learned model compresses the environment into a compact, predictable latent space where planning can occur efficiently.

Instead of interacting with the real environment for policy training, Dreamer's actor and critic networks are trained entirely on synthetic trajectories generated by the world model. This process, called latent imagination, involves:

• Rolling out imagined trajectories in the compact latent state space.
• Using predicted rewards from the world model to compute returns.
• Propagating gradients from the actor and critic networks back through the learned dynamics of the world model. This makes training sample-efficient and computationally cheaper than model-free alternatives, as it avoids expensive real-world steps.

Dreamer uses an actor-critic architecture for control, both trained on imagined rollouts.

• Critic (Value Model): Estimates the expected future return (value) of a given latent state. It is trained to regress the λ-return, a multi-step value target that balances bias and variance.
• Actor (Policy Model): Outputs a sequence of actions that maximize the predicted value from the critic. It is trained using the REINFORCE algorithm with a reparameterization trick, where gradients are backpropagated from the critic's value estimates, through the imagined dynamics, to the actor's parameters. This setup enables long-horizon planning within the learned model.

Dreamer is an off-policy algorithm that relies on a standard experience replay buffer. This buffer stores sequences of real interactions (images, actions, rewards) with the environment.

• The buffer provides the data used to train the world model via maximum likelihood.
• It decouples the data collection process from the learning process, improving sample efficiency.
• The world model is trained to accurately predict the dynamics and rewards of the real data distribution stored in the buffer, which grounds the latent imagination in reality.

Dreamer introduced several key ideas that advanced model-based RL:

• End-to-End Joint Training: The world model, actor, and critic are trained jointly from pixels, without hand-designed features.
• Stochastic Latent States: The RSSM's stochastic units capture uncertainty about the future, improving robustness.
• Long-Horizon Gradient Propagation: Gradients flow from the policy objective back through hundreds of imagined time steps, enabling effective long-term planning.
• Sample Efficiency: By learning a model, it often requires 5-50x fewer environment interactions than leading model-free agents like PPO or SAC to solve tasks.

Subsequent versions refined the original Dreamer algorithm:

• DreamerV2: Introduced categorical latent variables and KL balancing for more stable world model training. It achieved superhuman performance on the Atari 2600 benchmark from pixels alone.
• DreamerV3: Scaled the architecture and introduced symlog predictions and free bits, enabling stable training across a wide range of environments (from classic control to Minecraft) without hyperparameter tuning. It demonstrated robust performance on tasks requiring long-term memory and reasoning. These evolutions solidified Dreamer as a leading model-based RL paradigm.

Dreamer's core is a latent dynamics model that learns to predict future states in a compressed, abstract representation. This model comprises:

• A representation encoder that maps image observations to stochastic latent states.
• A transition model that predicts the next latent state given the current state and action.
• A reward predictor and optional continuation predictor (for terminal states).

By operating in this learned latent space, Dreamer avoids the computational burden of pixel-level prediction, enabling efficient long-horizon imagination.

The agent's policy and value functions are trained not on costly real environment steps, but on synthetic trajectories imagined by the world model. This process involves:

• Rolling out latent trajectories by unrolling the dynamics model from a sampled latent state.
• Computing latent rewards and values using the learned predictors.
• Applying backpropagation through time (BPTT) to optimize the actor and critic networks using these imagined sequences.

This decouples policy learning from the environment, providing massive sample efficiency gains, often requiring 5-10x fewer environment interactions than model-free counterparts.

Dreamer uses a standard experience replay buffer to store sequences of real observations, actions, and rewards. This buffer serves two critical functions:

1. Training the World Model: Batches of short sequences are sampled from the buffer to train the representation, dynamics, and prediction networks via gradient descent.
2. Grounding Imagination: The latent states from which imagination rolls out are initially encoded from real data sampled from the replay buffer. This ensures the imagined trajectories are anchored in plausible regions of the state space.

This combination of learning from past data and planning within a learned model is a hallmark of its sample-efficient design.

Dreamer was designed to work directly from pixel inputs, such as images from robotic cameras or game screens. It overcomes the complexity of high-dimensional spaces through:

• Compression: The representation encoder distills images into a low-dimensional latent vector containing only task-relevant information.
• Abstraction: The latent state discards irrelevant pixel-level details (like background textures), focusing on the underlying state variables (like object positions).
• Stochasticity: Using a variational autoencoder (VAE)-style approach with a Gaussian latent state helps capture uncertainty and multimodality in predictions.

This makes it applicable to real-world domains where constructing a low-dimensional state vector is impractical.

Dreamer's architecture cleanly separates three learned components that are trained jointly:

• World Model (Dynamics): An RSSM (Recurrent State-Space Model) that combines deterministic and stochastic latent paths for robust long-term prediction.
• Actor (Policy): A neural network that outputs action distributions, trained to maximize expected imagined value.
• Critic (Value Function): A neural network that estimates the expected return from a given latent state, trained via temporal difference learning on imagined trajectories.

The actor and critic are purely latent models, never seeing raw pixels during their training, which dramatically reduces their parameter count and training cost.

Dreamer provides several key benefits compared to model-free algorithms like DQN or PPO:

• Extreme Sample Efficiency: Learns complex behaviors with orders of magnitude fewer environment interactions.
• Safe Exploration: Can 'dream' of dangerous actions without executing them in the real world.
• Long-Horizon Credit Assignment: Backpropagation through imagined long trajectories provides clearer gradients for actions with delayed rewards.
• Data Reuse: The replay buffer allows the world model to be trained multiple times on each real experience.
• Generalization: The learned world model can often generalize to novel situations within the same domain, enabling zero-shot planning.

Its main trade-off is increased complexity in training the world model itself.

Latent imagination refers to the process of training a policy by rolling out trajectories within the compressed latent space of a learned world model, rather than interacting with the real environment. This is Dreamer's core innovation.

• The actor and critic networks are trained on these imagined sequences.
• It decouples policy learning from costly environment steps, dramatically improving sample efficiency.
• Training occurs on synthetic gradients computed from the model's predictions, enabling long-horizon credit assignment.

Model-Based Reinforcement Learning (MBRL) is a paradigm where an agent learns an explicit model of the environment's dynamics and uses it for planning or policy optimization. Dreamer is a leading MBRL algorithm.

• Contrasts with model-free RL (e.g., DQN, PPO), which learns a policy or value function directly from experience.
• Key advantages include superior sample efficiency and the potential for offline planning.
• The primary challenge is model bias; inaccuracies in the learned model can compound and lead to poor policies.

An experience replay buffer is a FIFO data structure that stores past experience tuples (state, action, reward, next state). Dreamer uses it to train its world model.

• Stores real environment interactions (image sequences).
• Enables batch sampling to break temporal correlations in the data, stabilizing world model training.
• Its capacity determines the diversity and recency of data used for learning the environment dynamics.

The Recurrent State-Space Model is the specific world model architecture at the heart of Dreamer. It factorizes the latent state into a deterministic and a stochastic component.

• Deterministic path: An RNN (like a GRU) that maintains a hidden state for temporal coherence.
• Stochastic path: A variable that captures uncertainty about the present and future.
• This factorization allows the model to robustly handle partial observability and generate diverse, plausible futures for imagination.