Model-Based Reinforcement Learning

The dynamics model is the core predictive component in MBRL. It is a function, often a neural network, that approximates the environment's transition function: s_{t+1} = f(s_t, a_t). It predicts the next state (and sometimes the reward) given the current state and action.

• Types: Can be deterministic (single prediction) or stochastic (predicts a distribution).
• Training: Typically learned via supervised regression on collected experience tuples (s_t, a_t, s_{t+1}).
• Role: Enables internal simulation (planning) without interacting with the real environment, improving data efficiency.

A planning algorithm uses the learned dynamics model to simulate potential future trajectories and select optimal actions. It searches over action sequences to maximize predicted cumulative reward.

• Common Methods: Include Monte Carlo Tree Search (MCTS) for discrete actions and Model Predictive Control (MPC) or random shooting for continuous control.
• Process: At each timestep, the algorithm runs multiple simulated rollouts from the current state, evaluates them using the model, and executes the first action of the best-found sequence.
• Benefit: Allows the agent to reason about long-term consequences before acting, leading to more deliberate behavior.

MBRL requires a strategy for gathering real-world experience to train the dynamics model and policy. This involves a critical balance between exploration and exploitation.

• Model-Based Exploration: The agent can use its model's uncertainty estimates to guide exploration towards poorly modeled states (e.g., via Bayesian neural networks or ensemble disagreement).
• Iterative Loop: A standard loop is: 1) Collect data using the current policy/model, 2) Train/improve the dynamics model on the aggregated dataset, 3) Improve the policy via planning or policy optimization within the model.
• Dilemma: Must gather data informative for improving the model, not just for maximizing short-term reward.

In MBRL, the policy—the function mapping states to actions—can be represented in two primary ways.

• Explicit Policy Network: A separate neural network is trained, often using gradients from planning or by distilling a planner's behavior. This enables fast, reactive execution.
• Implicit via Planning: The policy is defined on-the-fly by the planning algorithm (e.g., MPC). There is no separate policy network; action selection requires running the planner at every step.
• Hybrid Approaches: Many systems use an explicit policy network for efficiency but periodically refine it using planning-based targets, a process known as policy distillation.

A key challenge in MBRL is the distributional shift between states visited by the current policy and those the dynamics model was trained on. This leads to model exploitation where the policy finds actions that the model predicts are good, but which fail in the real environment.

• Mitigation Strategies:
  • Ensemble Models: Using multiple models and planning pessimistically (e.g., using the worst-case prediction) increases robustness.
  • Uncertainty-Aware Planning: Penalizing actions that lead to states where the model is uncertain.
  • Data Aggregation: Continuously adding new on-policy data to the training set to keep the model accurate for the current policy's state distribution.

While not always required, many advanced MBRL algorithms incorporate a value function or critic to improve planning efficiency and policy learning.

• Role of the Critic: A learned value function V(s) or action-value function Q(s,a) provides a fast estimate of long-term return from a state, reducing the depth of planning needed.
• Model-Based Value Expansion: Short model-based rollouts can be used to generate more accurate targets for training the value function, a technique used in algorithms like Model-Based Value Expansion (MVE).
• Actor-Critic Frameworks: MBRL can be integrated into actor-critic methods, where the model generates synthetic experience to train the critic, which in turn guides policy (actor) updates.

The foundational MBRL framework that interleaves real experience with simulated experience from a learned model. The agent:

• Learns a model from real interactions (state, action, next state).
• Plans by generating simulated rollouts from the model.
• Learns a value function/policy from both real and simulated data. This hybrid approach dramatically improves sample efficiency compared to purely model-free methods.

A receding-horizon optimal control method that uses a dynamics model for short-term planning. At each timestep, the agent:

1. Samples candidate action sequences.
2. Rolls out trajectories using the learned model.
3. Selects the first action of the sequence with the highest predicted reward.
4. Executes the action and re-plans from the new state. MPC is robust to model inaccuracies due to frequent re-planning and is widely used in robotics.

A Bayesian, data-efficient MBRL algorithm that explicitly models uncertainty. Key features:

• Uses Gaussian Processes (GPs) to learn a probabilistic dynamics model.
• Propagates uncertainty through multi-step predictions.
• Directly optimizes a policy to maximize expected long-term reward using gradient-based methods. PILCO is notable for learning complex control tasks with very few real-world trials (e.g., ~20 episodes).

A paradigm where an agent learns a generative model of its environment, often using latent variable models. The architecture typically includes:

• A Vision Model (V) that compresses high-dimensional observations into a latent space.
• A Memory/MDN-RNN (M) that predicts future latent states and rewards.
• A Controller (C) that learns to act using only the compact latent representations. This separation allows for fast planning in a learned, abstract latent space.

A state-of-the-art algorithm that learns a model without being given the rules of the environment. It jointly learns:

• A representation function to map observations to hidden states.
• A dynamics function to predict future hidden states and rewards.
• A prediction function for policy and value. MuZero performs Monte Carlo Tree Search (MCTS) within its learned hidden state model, achieving superhuman performance in games like Go, Chess, and Atari without any prior knowledge.

A latent world model algorithm that learns behaviors purely from imagined trajectories. Its components are:

• An encoder that compresses images to latent states.
• An RSSM (Recurrent State-Space Model) for predicting future latent states and rewards.
• A decoder for image reconstruction.
• An actor-critic trained entirely on latent imagination via backpropagation through time. Dreamer demonstrates highly sample-efficient learning from pixels.

A World Model is a learned, compact representation of an environment's dynamics. It is the core predictive engine in MBRL, enabling agents to simulate future states and rewards without interacting with the real environment.

• Function: Internally simulates the consequences of potential actions.
• Architecture: Often implemented as a recurrent neural network or latent dynamics model.
• Use Case: In Dreamer and related algorithms, the world model is used to train the policy entirely within its own generated latent imagination, dramatically improving sample efficiency.

Model-Free Reinforcement Learning is the dominant alternative paradigm where an agent learns a policy or value function directly from environment interaction, without explicitly learning or using a dynamics model.

• Key Distinction: No internal model of the environment's transition function.
• Algorithms: Includes Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Q-Networks (DQN).
• Trade-off: Typically more sample-inefficient than MBRL but avoids the complexity and potential inaccuracies of learning a model. MBRL often aims to match the asymptotic performance of model-free methods with far fewer real-world samples.

Model Predictive Control is a classical control technique that shares a core principle with MBRL: using a model for planning. At each timestep, MPC solves a finite-horizon optimization problem using the model, executes the first planned action, and then re-plans.

• Online Planning: Unlike a learned neural network policy, MPC re-optimizes the action sequence at every step.
• Model Use: Can use either an analytical physics model or a learned neural network model.
• Application: Widely used in robotics and process control. In MBRL, MPC is often the 'planner' component that leverages the learned dynamics model.

A Forward Dynamics Model is a specific type of world model that predicts the next state (s_{t+1}) given the current state (s_t) and action (a_t). It is the most common model learned in MBRL.

• Mathematical Form: f(s_t, a_t) -> s_{t+1}.
• Training: Typically trained via supervised learning on collected transition data (s_t, a_t, s_{t+1}).
• Challenge: Model Bias and Compounding Error—small prediction errors can accumulate over long simulated trajectories, leading the planner astray. Techniques like ensemble models are used to quantify uncertainty and mitigate this.

The Dyna Architecture is a seminal hybrid framework that blends model-free and model-based learning. The agent interleaves real experience with simulated experience from its learned model to update its policy or value function.

• Three Loops: 1) Real interaction (learning model & policy), 2) Model learning, 3) Planning (simulating experience from the model to improve the policy).
• Impact: Established the conceptual blueprint for modern MBRL algorithms that use a model to generate synthetic data for policy training.
• Efficiency: Dramatically improves data efficiency by reusing real data through model-based imagination.

An Inverse Dynamics Model predicts the action (a_t) that caused a transition from one state to another (s_t -> s_{t+1}). It is less common than forward models in MBRL but plays key roles in other areas.

• Mathematical Form: g(s_t, s_{t+1}) -> a_t.
• Primary Use: In Imitation Learning and robotics, for inferring actions from observed state sequences (e.g., from video demonstrations).
• Contrast with Forward Model: While a forward model predicts consequences, an inverse model infers the 'cause' (action) given a desired effect (state change). It can be used for planning via inverse shooting methods.