Model Error

The most critical property of model error is that it compounds multiplicatively over the course of an imagined rollout. A small single-step error ε can lead to a final state error on the order of ε^H, where H is the planning horizon. This exponential blow-up renders long-horizon planning with an imperfect model highly unreliable and is the primary motivation for short-horizon model usage in algorithms like Model Predictive Control (MPC).

Model error is rarely uniform. It is highly dependent on the state-action region being queried. Errors are typically lower in areas of the state space well-covered by the agent's training data and higher in novel or under-explored regions. This dependency forms the basis for uncertainty-aware exploration, where the agent actively seeks out high-error states to improve its model.

• In-Distribution: Low error, reliable for planning.
• Out-of-Distribution (OOD): High error, risky for exploitation.

Model error can be decomposed into systematic bias and stochastic variance.

• Systematic Bias: Consistent directional error caused by model misspecification (e.g., wrong network architecture) or distributional shift. Leads to model-policy co-adaptation, where a policy learns to exploit the model's flawed physics.
• Stochastic Variance: Non-deterministic error due to limited data or inherent environmental stochasticity. This is aleatoric uncertainty and cannot be reduced with more data, only better quantified. Managing this decomposition is key to robust uncertainty quantification.

When a policy is trained on synthetic data from a flawed model, it suffers from distributional shift. The policy becomes proficient in the "model environment," which diverges from reality. This manifests in two failure modes:

1. Exploitative Overfitting: The policy finds actions that yield high predicted reward in the model but fail or are catastrophic in the real environment.
2. Conservative Underperformance: In pessimistic exploration or offline RL, excessive penalization for model uncertainty can lead to overly cautious, suboptimal policies. Algorithms like MoDel-Based Policy Optimization (MBPO) deliberately use short rollouts to mitigate this.

Accurately measuring model error is essential for robust planning. Common technical approaches include:

• Probabilistic Ensembles: Train multiple models; use their disagreement (ensemble variance) as a proxy for epistemic uncertainty (model error).
• Bayesian Neural Networks (BNNs): Maintain weight distributions, providing principled predictive uncertainty.
• Calibration Metrics: Check if the model's predicted confidence intervals (e.g., 90% credible interval) actually contain the true outcome 90% of the time. A miscalibrated model is dangerous for certainty-equivalence control.

Advanced MBRL algorithms incorporate specific mechanisms to manage model error:

• Short-Horizon Planning (MPC): Limits compounding by frequently re-planning from fresh, real observations.
• Uncertainty-Weighted Trajectories: In planning, down-weight or discard trajectories predicted with high uncertainty.
• Latent Dynamics Models: Algorithms like Dreamer learn models in a compressed latent space, which can filter irrelevant noise and improve generalization, reducing error.
• Value-Equivalent Models: As in MuZero, learn a model that is only accurate for predicting future values and policies, not necessarily raw states, which can be a more forgiving target.

Compounding error is the phenomenon where inaccuracies in a learned dynamics model accumulate multiplicatively over the course of a multi-step imagined rollout. Small prediction errors at each step are fed back as input for the next prediction, leading to a rapid divergence from realistic simulated states. This is a primary failure mode in long-horizon planning and necessitates techniques like short rollouts, uncertainty-aware planning, and periodic re-sampling from the real environment.

Uncertainty quantification involves estimating the predictive uncertainty of a learned dynamics model, distinguishing between aleatoric uncertainty (inherent environmental stochasticity) and epistemic uncertainty (model ignorance due to limited data). Accurate quantification is critical for mitigating model error. Methods include:

• Bayesian Neural Networks (BNNs) for weight distribution.
• Probabilistic Ensembles where disagreement indicates uncertainty.
• Gaussian Processes for non-parametric uncertainty. These estimates guide pessimistic exploration and robust Model Predictive Control (MPC).

Model-policy co-adaptation is a pathological failure mode where a policy overfits to the specific biases and inaccuracies of its own learned dynamics model. The policy learns to exploit flaws in the model's simulation, achieving high reward in imagination but performing poorly when executed in the real environment. This highlights that reducing model error on a held-out test set does not guarantee a useful model for control. Mitigation strategies include regularizing the policy, using an ensemble of models, or employing value-equivalent models that prioritize planning accuracy over perfect dynamics prediction.

Certainty-equivalence control is a naive planning approach where an agent acts as if its learned dynamics model is perfectly accurate, completely ignoring predictive uncertainty. It simply solves for optimal actions using the model's mean predictions. This method is computationally simple but highly susceptible to model error, often leading to catastrophic failures when the agent encounters states where the model is inaccurate or has never been trained. Robust alternatives incorporate uncertainty explicitly, such as in Pessimistic Exploration or algorithms that plan under worst-case assumptions within an uncertainty set.

A value equivalent model is a learned model that is accurate only for the purpose of computing optimal values and policies, rather than needing to match the true environment's state transitions exactly. Pioneered by the MuZero algorithm, this paradigm redefines the objective of model learning. Instead of minimizing model error on state predictions, the model is trained to accurately predict future rewards, values, and policy distributions. This can be more sample-efficient and robust, as it avoids wasting capacity on modeling environment details irrelevant for decision-making.

System identification is the classical control theory process of learning a mathematical model of a system's dynamics from observed input-output data. In the context of model-based RL, it is the foundational step of learning the transition model. While traditional methods often assume linear or simple parametric forms, modern MBRL uses deep neural networks to identify complex, non-linear dynamics. The core challenge remains the same: minimizing model error on a dataset of interactions (s_t, a_t, s_{t+1}, r_t). This field provides well-established metrics and techniques for evaluating model fidelity.