Uncertainty Quantification

A probabilistic ensemble is a practical and highly effective method for estimating predictive uncertainty. It involves training multiple (e.g., 5-10) neural network dynamics models on the same dataset. The key insight is that the disagreement (variance) among the ensemble members' predictions for a given state-action pair serves as a proxy for epistemic uncertainty.

• Planning: Algorithms like PETS use ensemble disagreement to implement optimism in the face of uncertainty, favoring actions where the model ensemble disagrees, leading to active exploration.
• Robustness: In offline RL, a pessimistic policy can be trained by penalizing actions that lead to states with high ensemble variance.

Compounding error is the Achilles' heel of MBRL. Small inaccuracies in a single-step prediction are magnified when the model is unrolled over multiple time steps for planning. A 2% error per step can lead to a completely nonsensical predicted state after 50 steps. This phenomenon fundamentally limits the effective planning horizon. Agents must balance using longer horizons for better long-term decisions against the risk of planning based on increasingly unrealistic simulated states. Techniques like short-horizon model-based rollouts (used in MBPO) and replanning at every step (as in MPC) are direct engineering responses to this challenge.

Quantified uncertainty directly informs the exploration-exploitation trade-off.

• Model-Based Exploration: The agent intentionally seeks out states and actions where its dynamics model is most uncertain (high epistemic uncertainty). This is often implemented by adding an exploration bonus to the reward function proportional to the model's prediction variance, guiding the agent to regions that will most improve the model.
• Pessimistic Exploitation (Offline RL): When learning from a fixed dataset with no online interaction (offline RL), the agent must avoid exploiting spurious correlations in the model. Here, high uncertainty is treated as a danger signal. The policy is constrained or penalized for taking actions that lead to uncertain state predictions, preventing catastrophic failure due to model bias.

For high-dimensional observations (e.g., pixels), learning a dynamics model directly in observation space is inefficient. Latent world models (e.g., the Recurrent State-Space Model (RSSM) in Dreamer) learn to predict in a compressed, abstract latent space. Uncertainty quantification in these models operates in this latent space. The stochastic component of the RSSM captures aleatoric uncertainty, while techniques like latent ensemble or dropout can estimate epistemic uncertainty. Planning via latent imagination then uses these uncertainty estimates to generate robust behaviors from pixels, a cornerstone of modern sample-efficient MBRL.

A Bayesian Neural Network (BNN) is a neural network that represents its weights as probability distributions rather than single point estimates, providing a principled framework for uncertainty estimation.

• Mechanism: Instead of learning a fixed weight w, it learns a distribution p(w|Data). Predictions are made by integrating over all possible weights (Bayesian model averaging).
• Uncertainty Output: The variance of the predictive distribution naturally captures both epistemic and aleatoric uncertainty.
• Computational Trade-off: Exact inference is intractable; approximations like Variational Inference or Monte Carlo Dropout are used. This makes BNNs more computationally expensive than deterministic networks but valuable for robust dynamics modeling.

A probabilistic ensemble is a practical and highly effective method for uncertainty quantification, consisting of multiple neural networks trained to model the same dynamics.

• How it Works: An ensemble of N models (e.g., 5-10) is trained on the same data with different random initializations or bootstrapped datasets. Each model provides a prediction (μ_i, σ_i) for the next state.
• Uncertainty as Disagreement: The mean of the ensemble's predictions is often more accurate. The variance (disagreement between models) provides a strong signal for epistemic uncertainty.
• Use in MBRL: Used in algorithms like PETS and MBPO. The planner can then be uncertainty-aware, e.g., by optimizing a pessimistic objective (reward - β * uncertainty) to avoid model-exploitative paths.

Pessimistic exploration (or conservative model-based RL) is a strategy, crucial for offline RL and safe online learning, where an agent's policy is constrained to avoid states where its model is highly uncertain.

• Core Principle: Underestimates the value (or overestimates the cost) of actions leading to uncertain state transitions. This prevents the agent from exploiting model error.
• Implementation: Often achieved by subtracting a penalty proportional to the model's predictive uncertainty (e.g., ensemble variance) from the reward during planning.
• Trade-off: Introduces a robustness vs. performance trade-off. An overly pessimistic agent may fail to discover truly high-reward regions that are novel but safe.

Certainty-equivalence control is a naive planning baseline that treats a learned dynamics model as if it were perfectly accurate, completely ignoring predictive uncertainty.

• The Default Approach: Many simple MBRL implementations use this by default, employing a deterministic model and planning with standard optimizers.
• The Risk: This approach is highly susceptible to model exploitation and compounding error. The agent may confidently pursue a trajectory that is optimal in the flawed model but disastrous in reality.
• Contrast with UQ: Serves as a counterpoint to highlight the necessity of uncertainty quantification. Performance gaps between certainty-equivalence and uncertainty-aware methods reveal the cost of ignoring model error.