Model-Based Exploration

The core mechanism where the agent's internal dynamics model estimates its own predictive uncertainty. The agent is incentivized to explore states and actions where this uncertainty is high. Common techniques include:

• Upper Confidence Bound (UCB) applied to model predictions.
• Thompson Sampling, where actions are chosen based on a model sampled from a posterior distribution.
• Prediction Error as an intrinsic reward, where higher error signals more informative states.

A Bayesian Neural Network (BNN) represents network weights as probability distributions rather than single values. This provides a principled, mathematical framework for uncertainty quantification. During exploration:

• The BNN's posterior distribution over weights yields a distribution over possible next states.
• The variance of this distribution represents epistemic uncertainty (model uncertainty).
• Agents can target states that maximize this variance, ensuring the model learns from the most informative data.

A practical and highly effective method where an ensemble of multiple neural networks is trained on the same transition data. For exploration:

• Each network in the ensemble makes a prediction for the next state given a state-action pair.
• Disagreement among the ensemble members is used as a proxy for model uncertainty.
• The agent explores trajectories where ensemble disagreement is maximal. This approach is more computationally tractable than full BNNs and is a cornerstone of algorithms like PETS (Probabilistic Ensembles with Trajectory Sampling).

A classic exploration principle adapted for model-based settings. The agent's planning algorithm is biased to be optimistic about the outcomes in uncertain regions. Mechanically:

• The learned model produces not just a predicted next state, but an optimistic estimate (e.g., predicted reward + an uncertainty bonus).
• Planning (e.g., via Monte Carlo Tree Search) then favors paths through these optimistic, uncertain regions.
• This leads to systematic exploration of the state-action space while still aiming for high reward, balancing the exploration-exploitation trade-off.

Exploration is directed not randomly, but towards achieving specified goals. The agent uses its model to predict which actions will reduce the prediction error relative to a goal state.

• A forward dynamics model predicts the next state.
• An inverse dynamics model may predict the action taken.
• Disagreement between these models, or high error in reaching a predicted goal state, generates an intrinsic curiosity reward.
• This creates a self-supervised loop where the agent seeks out experiences that maximize learning progress towards diverse goals.

A provably efficient, Bayesian model-based exploration algorithm. The agent maintains a posterior distribution over possible Markov Decision Processes (MDPs) that could describe the true environment.

• At the start of each episode, the agent samples a single MDP from this posterior.
• It then acts optimally with respect to the sampled MDP for the duration of the episode (exploiting the sample).
• By repeatedly sampling and acting optimally, the agent automatically explores in proportion to its uncertainty, without needing explicit uncertainty bonuses.

Model-Based Reinforcement Learning (MBRL) is a paradigm where an agent learns an internal model of its environment's dynamics and reward function. This model is then used for planning and policy optimization, with the primary goal of achieving greater sample efficiency compared to model-free methods.

• Core Idea: Replace or supplement expensive real-world trials with cheap internal simulations.
• Key Components: A dynamics (transition) model and a reward model.
• Primary Use Case: Applications where real-world interaction is costly, risky, or slow, such as robotics or autonomous driving.

A world model is an agent's internal, learned representation that predicts future states and rewards based on current states and actions. It serves as a compressed simulator, enabling planning and imagination without direct interaction with the real environment.

• Function: Encodes the agent's understanding of "how the world works."
• Architecture: Often implemented as a latent dynamics model (e.g., a Recurrent State-Space Model) to handle high-dimensional observations like images.
• Example: In the Dreamer algorithm, the agent learns a world model and then trains its policy entirely through latent imagination within this model.

Uncertainty quantification in model-based RL involves estimating the epistemic (model) uncertainty and aleatoric (environmental) uncertainty in a learned dynamics model's predictions. This is critical for robust planning and guiding exploration.

• Why it Matters: Planning with an overconfident, inaccurate model leads to poor performance. Uncertainty estimates tell the agent where its model is unreliable.
• Methods for Estimation:
  • Bayesian Neural Networks (BNNs): Represent network weights as probability distributions.
  • Probabilistic Ensembles: Use disagreement among multiple neural networks as a proxy for uncertainty.
• Exploitation: In pessimistic exploration, the agent is penalized for acting in high-uncertainty states.

Model Predictive Control (MPC) is an online planning algorithm used in model-based reinforcement learning. At each step, it solves a finite-horizon optimal control problem using the learned model, executes only the first action from the planned sequence, and then replans from the new state.

• Key Feature: Receding horizon control – constantly re-optimizing based on new observations.
• Advantage: Naturally robust to small model errors because it frequently re-synchronizes with the real world.
• Computational Trade-off: Requires solving an optimization problem at every timestep, which can be expensive for long planning horizons or complex models.

Sample efficiency refers to the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the primary claimed advantage of model-based over model-free reinforcement learning.

• Model-Free RL: Often requires millions or billions of environment steps to learn, which is impractical for real robots or expensive simulators.
• Model-Based RL: Aims to learn a usable model from a relatively small dataset, then use that model to generate vast amounts of imagined rollouts for policy training.
• Metric: Measured by the final policy performance achieved after a fixed, small number of real environment steps.

Compounding error is a fundamental challenge in model-based RL where inaccuracies in a learned dynamics model accumulate over the course of a multi-step imagined rollout. Small prediction errors at each step lead to increasingly unrealistic and divergent simulated states.

• Consequence: A policy trained on long, erroneous rollouts will fail in the real environment.
• Mitigation Strategies:
  • Using short rollouts for training (as in MBPO).
  • Uncertainty-aware planning that avoids uncertain state trajectories.
  • Learning value-equivalent models (like in MuZero) that are accurate for planning but not necessarily for exact state prediction.
• This phenomenon directly motivates the need for intelligent, uncertainty-driven exploration.