Latent Dynamics Model

The encoder is a neural network (typically a Convolutional Neural Network for images) that maps raw, high-dimensional observations (e.g., pixels) into a low-dimensional latent state vector z_t. This compression discards irrelevant details (like background noise) while preserving the information necessary for predicting future states. It transforms the pixel space into a more tractable representation space for learning dynamics.

• Function: z_t = encoder(o_t)
• Purpose: Dimensionality reduction and feature extraction.
• Example: In a robot arm task, the encoder learns to represent the positions and velocities of joints from camera images, ignoring lighting variations.

The core transition model is a learned function (often a recurrent neural network like an LSTM or GRU) that predicts the next latent state given the current one and an action. It defines the learned latent dynamics: z_{t+1} = transition(z_t, a_t). This model operates entirely in the latent space, making predictions computationally efficient compared to predicting raw pixels.

• Key Challenge: Avoiding compounding error, where small prediction mistakes accumulate over long imagined sequences.
• Architectures: May be deterministic (single prediction) or stochastic (predicts a distribution, e.g., using a Bayesian Neural Network), with the latter better at capturing uncertainty.

The decoder is a generative network (often a transposed CNN) that maps a latent state z_t back to a reconstruction of the original observation o_t. It is trained alongside the encoder via a reconstruction loss (e.g., mean squared error). Its primary role is to ensure the latent space retains meaningful information about the observation. For planning, the decoder may also be used to predict reward signals r_t or task-relevant features (like "game score") directly from the latent state.

• Function: ô_t, ȓ_t = decoder(z_t)
• Purpose: Validates the latent representation's fidelity and enables reward prediction.

A sophisticated and common architecture for latent dynamics models is the Recurrent State-Space Model (RSSM), used in algorithms like Dreamer. It explicitly separates latent state into:

• Deterministic state (h_t): Managed by an RNN to track temporal dependencies.
• Stochastic state (z_t): A random variable capturing unpredictable aspects of the future.

The transition is: h_t = RNN(h_{t-1}, z_{t-1}, a_{t-1}) and z_t ~ distribution( h_t ). This hybrid design improves long-term sequence modeling and uncertainty estimation, making it highly effective for imagined rollouts.

This is not part of the learned model itself but is the primary consumer of it. Using the learned latent dynamics, an agent can perform planning by running imagined rollouts (or dreams). Starting from an encoded state, it uses the transition model to simulate multiple potential future trajectories in latent space, evaluating them with the decoded reward predictions. Algorithms like Model Predictive Control (MPC) or policy optimization via backpropagation through time are used to select optimal actions. This enables decision-making without interacting with the slower, real environment.

Critical for robust planning, this component quantifies the model's confidence in its predictions. Common implementations include:

• Probabilistic Ensembles: Training multiple transition models; their disagreement indicates epistemic uncertainty.
• Bayesian Neural Networks: Representing network weights as distributions.
• Stochastic Latent Variables: As in the RSSM, where the variance of z_t's distribution reflects uncertainty.

This uncertainty is used for pessimistic exploration (avoiding unfamiliar states) or uncertainty-aware planning, preventing the agent from exploiting model flaws, a failure mode known as model-policy co-adaptation.

A world model is an agent's internal, learned representation that predicts future environment states and rewards. It enables planning and imagination without direct, costly interaction with the real world. In MBRL, a latent dynamics model is a specific type of world model that operates in a compressed latent space.

• Core Function: Serves as a surrogate simulator for the agent.
• Example: In the Dreamer algorithm, the world model is a Recurrent State-Space Model (RSSM) that predicts future latent states.

Model Predictive Control (MPC) is an online planning algorithm that uses a learned dynamics model (like a latent dynamics model) for short-horizon, receding-horizon control. At each step, it plans an optimal sequence of actions, executes only the first, then re-plans from the new state.

• Key Feature: Robust to model inaccuracies due to frequent re-planning.
• Use Case: Common in robotics and process control where the model is approximate but the planning horizon is short.

Compounding error is a critical challenge in MBRL where small inaccuracies in a learned dynamics model accumulate over the course of a multi-step imagined rollout. This leads the agent's internal simulation to diverge into unrealistic states, causing planning failures.

• Cause: Imperfect model generalization or insufficient training data.
• Mitigation: Techniques include using short planning horizons (as in MPC), uncertainty quantification, and training policies on a mixture of real and short simulated rollouts.

Dreamer is a seminal model-based reinforcement learning algorithm that trains agents entirely in latent space. It learns a latent dynamics model (an RSSM) and then uses latent imagination—backpropagating through time on imagined rollouts—to train a policy and value function.

• Key Innovation: Achieves high sample efficiency by avoiding costly pixel-level prediction.
• Result: The policy learns from years of simulated experience generated in seconds of compute.

Uncertainty quantification involves estimating the epistemic (model) and aleatoric (environment stochasticity) uncertainty in a learned dynamics model's predictions. This is essential for robust planning and safe exploration.

• Methods: Bayesian Neural Networks (BNNs) and probabilistic ensembles are common approaches.
• Application in Planning: Algorithms can use uncertainty estimates to avoid exploiting areas where the model is unreliable (pessimistic exploration).

Sample efficiency measures the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the primary motivation for model-based RL. Latent dynamics models enhance sample efficiency by allowing the agent to learn from compact, abstract simulations.

• Contrast: Model-free RL (e.g., PPO, DQN) typically requires orders of magnitude more environment samples.
• Trade-off: Improved sample efficiency often comes at the cost of increased computational complexity for model learning and planning.