Latent Dynamics Model

The decoder is a generative network (often a transposed CNN) that maps a latent state z_t back to a reconstruction of the original observation o_t. It is trained alongside the encoder via a reconstruction loss (e.g., mean squared error). Its primary role is to ensure the latent space retains meaningful information about the observation. For planning, the decoder may also be used to predict reward signals r_t or task-relevant features (like "game score") directly from the latent state.

• Function: ô_t, ȓ_t = decoder(z_t)
• Purpose: Validates the latent representation's fidelity and enables reward prediction.

A sophisticated and common architecture for latent dynamics models is the Recurrent State-Space Model (RSSM), used in algorithms like Dreamer. It explicitly separates latent state into:

• Deterministic state (h_t): Managed by an RNN to track temporal dependencies.
• Stochastic state (z_t): A random variable capturing unpredictable aspects of the future.

The transition is: h_t = RNN(h_{t-1}, z_{t-1}, a_{t-1}) and z_t ~ distribution( h_t ). This hybrid design improves long-term sequence modeling and uncertainty estimation, making it highly effective for imagined rollouts.

This is not part of the learned model itself but is the primary consumer of it. Using the learned latent dynamics, an agent can perform planning by running imagined rollouts (or dreams). Starting from an encoded state, it uses the transition model to simulate multiple potential future trajectories in latent space, evaluating them with the decoded reward predictions. Algorithms like Model Predictive Control (MPC) or policy optimization via backpropagation through time are used to select optimal actions. This enables decision-making without interacting with the slower, real environment.

Critical for robust planning, this component quantifies the model's confidence in its predictions. Common implementations include:

• Probabilistic Ensembles: Training multiple transition models; their disagreement indicates epistemic uncertainty.
• Bayesian Neural Networks: Representing network weights as distributions.
• Stochastic Latent Variables: As in the RSSM, where the variance of z_t's distribution reflects uncertainty.

This uncertainty is used for pessimistic exploration (avoiding unfamiliar states) or uncertainty-aware planning, preventing the agent from exploiting model flaws, a failure mode known as model-policy co-adaptation.

Compounding error is a critical challenge in MBRL where small inaccuracies in a learned dynamics model accumulate over the course of a multi-step imagined rollout. This leads the agent's internal simulation to diverge into unrealistic states, causing planning failures.

• Cause: Imperfect model generalization or insufficient training data.
• Mitigation: Techniques include using short planning horizons (as in MPC), uncertainty quantification, and training policies on a mixture of real and short simulated rollouts.

Dreamer is a seminal model-based reinforcement learning algorithm that trains agents entirely in latent space. It learns a latent dynamics model (an RSSM) and then uses latent imagination—backpropagating through time on imagined rollouts—to train a policy and value function.

• Key Innovation: Achieves high sample efficiency by avoiding costly pixel-level prediction.
• Result: The policy learns from years of simulated experience generated in seconds of compute.

Uncertainty quantification involves estimating the epistemic (model) and aleatoric (environment stochasticity) uncertainty in a learned dynamics model's predictions. This is essential for robust planning and safe exploration.

• Methods: Bayesian Neural Networks (BNNs) and probabilistic ensembles are common approaches.
• Application in Planning: Algorithms can use uncertainty estimates to avoid exploiting areas where the model is unreliable (pessimistic exploration).

Sample efficiency measures the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the primary motivation for model-based RL. Latent dynamics models enhance sample efficiency by allowing the agent to learn from compact, abstract simulations.

• Contrast: Model-free RL (e.g., PPO, DQN) typically requires orders of magnitude more environment samples.
• Trade-off: Improved sample efficiency often comes at the cost of increased computational complexity for model learning and planning.