Model-Based Reinforcement Learning (MBRL)

The dynamics model, or transition model, is the core predictive component. It is a learned function, typically a neural network, that approximates the environment's true transition function: s_{t+1} = f(s_t, a_t). Its accuracy is paramount, as errors compound during long-horizon rollouts. Common architectures include:

• Ensemble Probabilistic Networks: Multiple networks whose disagreement quantifies epistemic uncertainty.
• Latent Dynamics Models (e.g., RSSM): Encode high-dimensional observations (like images) into a compact latent state for efficient prediction.
• Bayesian Neural Networks (BNNs): Represent model weights as distributions to capture uncertainty.

The reward model is a learned function that predicts the expected scalar reward r_t for a given state-action pair (s_t, a_t). It allows the agent to evaluate the desirability of imagined futures without requiring an environment-provided reward at every simulated step. In some frameworks like MuZero, the reward model is learned jointly with the value and policy models as part of a value-equivalent model, where accuracy is prioritized only for improving decision-making, not for perfect reward prediction.

This component uses the learned models to generate action sequences. It performs trajectory optimization within the internal simulation. Key algorithms include:

• Model Predictive Control (MPC): An online planner that solves a finite-horizon optimization at each step, executes the first action, and replans.
• Monte Carlo Tree Search (MCTS): A heuristic search algorithm that builds a lookahead tree by sampling simulated trajectories, used famously in AlphaZero and MuZero.
• Trajectory Optimization: Methods like the Iterative Linear Quadratic Regulator (iLQR) that use gradient-based optimization to find optimal action sequences under the model.

While planning can be used online, many MBRL algorithms also learn an explicit policy (π(a|s)) and value function (V(s)) from imagined rollouts. This decouples the expensive planning process from rapid action selection. In Model-Based Policy Optimization (MBPO), short model-generated rollouts create synthetic experience to train a policy via standard model-free algorithms like SAC or PPO. The Dreamer algorithm trains both policy and value functions entirely through backpropagation in its latent world model.

A critical subsystem that estimates the reliability of the model's predictions. Model error is the primary failure point in MBRL. Quantifying uncertainty enables:

• Robust Planning: Algorithms can avoid actions leading to states where the model is uncertain (pessimistic exploration).
• Directed Exploration: Actively seeking states with high prediction error to gather data that improves the model.
• Managing Compounding Error: Limiting the planning horizon or trusting model predictions less over long simulated sequences.

This component manages the agent's real and synthetic experience. It typically maintains two buffers:

1. Real Experience Buffer: Stores state-action-reward-next_state tuples (s, a, r, s') from actual environment interaction.
2. Model (or Synthetic) Buffer: Stores trajectories generated via imagined rollouts from the dynamics model. The model is trained on data from the real buffer. The policy can be trained on data from both buffers, a process central to model-based offline RL where only a static dataset is available.

A world model is an agent's internal, learned representation that predicts future environment states and rewards based on current states and actions. It enables planning and imagination without direct, costly interaction with the real environment. This compressed, predictive representation is the foundational abstraction in MBRL.

• Function: Serves as a simulator for the agent.
• Output: Predicts next state and reward.
• Key Benefit: Enables sample-efficient learning through internal simulation.

Also called a dynamics model, a transition model is the core learned function within a world model. It specifically predicts the next state s_{t+1} given the current state s_t and action a_t. It encodes the agent's understanding of how its actions change the environment.

• Formal Definition: f_θ(s_t, a_t) → s_{t+1}.
• Primary Challenge: Model error—the discrepancy between predicted and true transitions.
• Architectures: Can be deterministic neural networks, probabilistic ensembles, or latent models for high-dimensional observations.

Model Predictive Control (MPC) is a dominant online planning algorithm in MBRL. At each step, it uses the learned model to simulate multiple potential action sequences over a finite planning horizon, selects the optimal sequence, but executes only the first action before replanning. This closed-loop approach is robust to model inaccuracies.

• Core Loop: Plan → Execute first action → Re-observe state → Re-plan.
• Advantage: Naturally handles constraints and model drift.
• Use Case: Common in robotics and process control where safety is critical.

Model-Based Policy Optimization (MBPO) is a hybrid algorithm that blends model-based and model-free RL. It uses short imagined rollouts from a learned dynamics model to generate large amounts of synthetic experience. This synthetic data is then used to train a policy using advanced model-free algorithms like Soft Actor-Critic (SAC) or PPO.

• Key Insight: Shorter rollouts minimize compounding error.
• Process: 1. Learn model. 2. Generate synthetic rollouts. 3. Train policy on mixed real/synthetic data.
• Result: Achieves high sample efficiency while leveraging powerful model-free optimizers.

Compounding error is a critical failure mode in MBRL where inaccuracies in a learned transition model accumulate over the course of a multi-step imagined rollout. Small prediction errors at each step are magnified, leading the model to simulate states that are increasingly unrealistic and far from the true environment distribution.

• Analogy: Similar to error propagation in numerical integration.
• Mitigation Strategies: Using short planning horizons (MPC), training on model-policy rollouts, employing probabilistic models that quantify uncertainty, and leveraging latent dynamics models that operate in a more stable abstract space.

Uncertainty quantification is the process of estimating the epistemic (model) and aleatoric (environmental stochasticity) uncertainty in a learned dynamics model's predictions. Accurate uncertainty estimates are essential for robust planning and intelligent exploration.

• Epistemic Uncertainty: Reduced with more data. Estimated via Bayesian Neural Networks (BNNs) or probabilistic ensembles.
• Aleatoric Uncertainty: Inherent randomness in the environment.
• Planning Use: Algorithms can avoid states with high epistemic uncertainty (pessimism) or explicitly seek them out for exploration.