Uncertainty Quantification

Uncertainty in MBRL is decomposed into two fundamental types. Epistemic uncertainty (or model uncertainty) arises from a lack of knowledge about the true environment dynamics and can be reduced by collecting more data. Aleatoric uncertainty (or environmental stochasticity) is inherent randomness in the system (e.g., sensor noise) and cannot be reduced with more data. Effective planning requires distinguishing between the two: epistemic uncertainty should guide exploration, while aleatoric uncertainty must be accounted for in robust control strategies.

A Bayesian Neural Network (BNN) provides a principled, probabilistic framework for uncertainty quantification. Instead of learning fixed weight values, a BNN learns a probability distribution over possible weights. This allows the model to express its confidence in its own predictions. For a given input, the BNN outputs a distribution of possible next states. The variance of this distribution quantifies the model's predictive uncertainty, which can be directly used for uncertainty-aware planning and risk-sensitive exploration.

A probabilistic ensemble is a practical and highly effective method for estimating predictive uncertainty. It involves training multiple (e.g., 5-10) neural network dynamics models on the same dataset. The key insight is that the disagreement (variance) among the ensemble members' predictions for a given state-action pair serves as a proxy for epistemic uncertainty.

• Planning: Algorithms like PETS use ensemble disagreement to implement optimism in the face of uncertainty, favoring actions where the model ensemble disagrees, leading to active exploration.
• Robustness: In offline RL, a pessimistic policy can be trained by penalizing actions that lead to states with high ensemble variance.

Compounding error is the Achilles' heel of MBRL. Small inaccuracies in a single-step prediction are magnified when the model is unrolled over multiple time steps for planning. A 2% error per step can lead to a completely nonsensical predicted state after 50 steps. This phenomenon fundamentally limits the effective planning horizon. Agents must balance using longer horizons for better long-term decisions against the risk of planning based on increasingly unrealistic simulated states. Techniques like short-horizon model-based rollouts (used in MBPO) and replanning at every step (as in MPC) are direct engineering responses to this challenge.

Quantified uncertainty directly informs the exploration-exploitation trade-off.

• Model-Based Exploration: The agent intentionally seeks out states and actions where its dynamics model is most uncertain (high epistemic uncertainty). This is often implemented by adding an exploration bonus to the reward function proportional to the model's prediction variance, guiding the agent to regions that will most improve the model.
• Pessimistic Exploitation (Offline RL): When learning from a fixed dataset with no online interaction (offline RL), the agent must avoid exploiting spurious correlations in the model. Here, high uncertainty is treated as a danger signal. The policy is constrained or penalized for taking actions that lead to uncertain state predictions, preventing catastrophic failure due to model bias.

For high-dimensional observations (e.g., pixels), learning a dynamics model directly in observation space is inefficient. Latent world models (e.g., the Recurrent State-Space Model (RSSM) in Dreamer) learn to predict in a compressed, abstract latent space. Uncertainty quantification in these models operates in this latent space. The stochastic component of the RSSM captures aleatoric uncertainty, while techniques like latent ensemble or dropout can estimate epistemic uncertainty. Planning via latent imagination then uses these uncertainty estimates to generate robust behaviors from pixels, a cornerstone of modern sample-efficient MBRL.

Model error is the discrepancy between the predictions of a learned dynamics model and the true environment dynamics. It is the fundamental quantity that uncertainty quantification aims to measure and mitigate.

• Primary Source of Failure: Unmanaged model error is the main cause of performance degradation in MBRL, as policies trained on inaccurate simulations fail in the real world.
• Decomposition: Often broken into epistemic uncertainty (reducible through more data) and aleatoric uncertainty (inherent stochasticity).
• Impact on Planning: High model error in critical state regions can lead the agent to pursue catastrophically suboptimal or dangerous imagined trajectories.

Compounding error is the phenomenon where inaccuracies in a learned dynamics model accumulate multiplicatively over the course of a multi-step imagined rollout.

• The Simulation Drift Problem: A small error in predicting step t+1 becomes the input for step t+2, leading to increasingly unrealistic and divergent simulated states far into the future.
• Limits Planning Horizon: This effect fundamentally limits the useful planning horizon, as long rollouts become untrustworthy.
• Mitigation Strategies: Techniques include using shorter rollouts (e.g., in MBPO), replanning frequently (as in MPC), and employing probabilistic models that can detect when predictions become unreliable.

A Bayesian Neural Network (BNN) is a neural network that represents its weights as probability distributions rather than single point estimates, providing a principled framework for uncertainty estimation.

• Mechanism: Instead of learning a fixed weight w, it learns a distribution p(w|Data). Predictions are made by integrating over all possible weights (Bayesian model averaging).
• Uncertainty Output: The variance of the predictive distribution naturally captures both epistemic and aleatoric uncertainty.
• Computational Trade-off: Exact inference is intractable; approximations like Variational Inference or Monte Carlo Dropout are used. This makes BNNs more computationally expensive than deterministic networks but valuable for robust dynamics modeling.

A probabilistic ensemble is a practical and highly effective method for uncertainty quantification, consisting of multiple neural networks trained to model the same dynamics.

• How it Works: An ensemble of N models (e.g., 5-10) is trained on the same data with different random initializations or bootstrapped datasets. Each model provides a prediction (μ_i, σ_i) for the next state.
• Uncertainty as Disagreement: The mean of the ensemble's predictions is often more accurate. The variance (disagreement between models) provides a strong signal for epistemic uncertainty.
• Use in MBRL: Used in algorithms like PETS and MBPO. The planner can then be uncertainty-aware, e.g., by optimizing a pessimistic objective (reward - β * uncertainty) to avoid model-exploitative paths.

Pessimistic exploration (or conservative model-based RL) is a strategy, crucial for offline RL and safe online learning, where an agent's policy is constrained to avoid states where its model is highly uncertain.

• Core Principle: Underestimates the value (or overestimates the cost) of actions leading to uncertain state transitions. This prevents the agent from exploiting model error.
• Implementation: Often achieved by subtracting a penalty proportional to the model's predictive uncertainty (e.g., ensemble variance) from the reward during planning.
• Trade-off: Introduces a robustness vs. performance trade-off. An overly pessimistic agent may fail to discover truly high-reward regions that are novel but safe.

Certainty-equivalence control is a naive planning baseline that treats a learned dynamics model as if it were perfectly accurate, completely ignoring predictive uncertainty.

• The Default Approach: Many simple MBRL implementations use this by default, employing a deterministic model and planning with standard optimizers.
• The Risk: This approach is highly susceptible to model exploitation and compounding error. The agent may confidently pursue a trajectory that is optimal in the flawed model but disastrous in reality.
• Contrast with UQ: Serves as a counterpoint to highlight the necessity of uncertainty quantification. Performance gaps between certainty-equivalence and uncertainty-aware methods reveal the cost of ignoring model error.