Model-Based Reinforcement Learning

The core of an MBRL system is a dynamics model—a function approximator (often a neural network) that predicts the next state and reward given the current state and action: s_{t+1}, r_t = f_θ(s_t, a_t). This model is typically trained via supervised learning on a dataset of real interactions (s, a, s', r). Common architectures include ensemble models (multiple networks) to estimate prediction uncertainty and recurrent networks for partially observable settings. The model's accuracy is critical; errors can compound during multi-step planning, leading to suboptimal or catastrophic policies.

Planning algorithms use the learned dynamics model as a simulator to evaluate sequences of potential actions. Instead of learning a direct policy, the agent plans online from its current state. Key methods include:

• Model Predictive Control (MPC): At each timestep, solves a finite-horizon optimization problem (e.g., via random shooting or Cross-Entropy Method) to select the best action sequence, executes the first action, and re-plans.
• Monte Carlo Tree Search (MCTS): Builds a look-ahead search tree by sampling trajectories through the model, balancing exploration and exploitation to find high-value paths.
• Trajectory Optimization: Uses gradient-based methods to optimize action sequences directly through the model.

In many MBRL frameworks, a policy network π_φ(a|s) and/or a value function V_ψ(s) are learned using data generated by the model. This serves two purposes:

1. Accelerating Planning: A value function provides a terminal cost estimate for truncated rollouts in MPC, improving long-horizon reasoning.
2. Providing a Fallback: A trained policy can act as a fast, reactive controller when computational budget for planning is limited. These components are typically trained with standard RL algorithms (like SAC or PPO), but using model-generated synthetic data ("model-based policy optimization"), which is far more sample-efficient than pure model-free learning.

A replay buffer stores all real interactions (s, a, r, s') the agent has experienced. It serves dual purposes:

• Training the Dynamics Model: Data is sampled from the buffer to continually improve model accuracy.
• Training the Policy/Value Networks: In hybrid approaches, data is used for policy optimization. A critical strategy is model-based data collection: the agent uses its current model and policy to decide which real-world actions to take, explicitly seeking to reduce model uncertainty (active learning or exploration) or maximize predicted reward. This creates a loop where better data leads to a better model, which enables better data collection.

The dynamics model functions as an internal simulator or world model. When the model is latent or imagined, it operates on compressed representations of states rather than raw pixels (e.g., using a variational autoencoder). This allows for extremely fast internal "dreaming" or imagination rollouts, where the agent can practice thousands of trials without touching the real environment. Frameworks like Dreamer exemplify this, where policies are trained entirely within the latent dynamics model via gradient backpropagation through imagined trajectories.

A defining feature of robust MBRL is explicit uncertainty quantification in the dynamics model. Since the model is always imperfect, especially in novel states, the system must know what it doesn't know. Techniques include:

• Ensemble Dynamics Models: Training multiple models; disagreement among their predictions serves as a proxy for epistemic uncertainty.
• Probabilistic Networks: Outputting a distribution (e.g., Gaussian) over next states. This uncertainty drives pessimistic planning (avoiding uncertain regions) or directed exploration (purposely visiting uncertain states to gather informative data), which is essential for safe and efficient real-world deployment.

The primary alternative to MBRL. In model-free RL, the agent learns a policy or value function directly from experience without constructing or using an explicit model of the environment's dynamics. Algorithms like Q-Learning, Policy Gradients, PPO, and SAC are model-free. They are often simpler to implement but typically require many more interactions with the environment (lower sample efficiency) compared to model-based approaches. Model-free methods are the foundation upon which many advanced RL algorithms are built.

The core learned component in MBRL. A dynamics model is a function (often a neural network) that predicts the next state and reward given the current state and action: (s', r) = f(s, a). It approximates the true environment transition function T(s, a). Types include:

• Deterministic models: Predict a single next state.
• Stochastic/probabilistic models: Predict a distribution over possible next states (e.g., using Gaussian outputs), which better captures uncertainty and can prevent model exploitation where the agent finds shortcuts in an inaccurate model. The fidelity of this model is the primary bottleneck for MBRL performance.

The process of using a model to simulate future trajectories and select optimal actions. In MBRL, once a dynamics model is learned, planning algorithms are used to "think ahead." Common methods include:

• Random Shooting: Simulate many random action sequences and execute the first action of the best one.
• Cross-Entropy Method (CEM): Iteratively refine a distribution over action sequences.
• Monte Carlo Tree Search (MCTS): Builds a search tree by selectively exploring promising sequences.
• Model Predictive Control (MPC): A receding-horizon control technique that plans a short sequence, executes the first action, then re-plans at the next step. This is highly effective for robotics due to its robustness to model errors.

A classic hybrid framework that blends model-free and model-based learning. In the Dyna architecture, the agent:

1. Interacts with the real environment (real experience).
2. Uses this experience to learn a model.
3. Uses the model to generate simulated experience.
4. Learns a policy/value function using both real and simulated experience via a model-free algorithm like Q-learning. This approach increases sample efficiency by leveraging the model as a source of additional, cheap training data. It exemplifies how a model can be used to augment rather than replace model-free learning.

A modern, deep learning-centric approach to MBRL where the dynamics model is a latent-space model. Popularized by the World Models paper, this involves learning a compressed latent representation of the state (via a Variational Autoencoder) and a recurrent model (like an RNN or Transformer) that predicts future latent states. The agent then learns a simple controller (policy) entirely within this learned, compact latent dream world. This decouples representation learning and dynamics prediction from control, often leading to more stable training and the ability to learn from pixels directly.

The key metric driving the use of MBRL, especially in robotics. Sample efficiency measures how many interactions with the real environment an agent requires to learn a competent policy. Robotics applications demand high sample efficiency because:

• Real-world interaction is slow (real-time physics).
• Real-world interaction is costly (wear and tear on hardware).
• Real-world interaction can be dangerous (an untrained robot can break itself or its surroundings). MBRL aims for high sample efficiency by learning a model from limited real data, then using it for extensive, free planning and simulation internally. This is contrasted with model-free methods that often require millions or billions of environment steps.