Compounding Error

The core promise of MBRL—high sample efficiency—is negated. If rollouts are too short to avoid compounding error, little useful synthetic data is generated. If they are too long, the synthetic data is corrupted and poisons policy training. Engineers must then fall back to costly real-environment interaction to correct the policy, erasing MBRL's primary advantage.

• Quantitative Impact: An algorithm like Model-Based Policy Optimization (MBPO) relies on short, accurate rollouts. Compounding error forces shorter horizons, reducing the value of each imagined rollout and requiring more real data collection.

Model Predictive Control (MPC), which replans at each step, is particularly vulnerable. While replanning mitigates error by correcting course, a severely inaccurate model means every new plan starts from a flawed belief state and compounds anew. The agent is perpetually 'chasing its tail,' leading to hesitant, oscillatory, or unstable behavior in the real environment.

• Real-World Effect: An autonomous vehicle using MPC with a poor dynamics model may exhibit jerky, over-corrective steering as each new plan based on faulty predictions leads to another unexpected state.

Compounding error corrupts uncertainty quantification. An agent cannot distinguish between states that are truly uncertain/novel and states that are simply miscalculated. This undermines pessimistic exploration strategies designed for safety. The agent may avoid safe, known states (due to imagined error) or confidently enter dangerous ones (due to unrealistically certain predictions).

• Link to Uncertainty: Methods using Probabilistic Ensembles or Bayesian Neural Networks to estimate uncertainty rely on the model's ability to self-assess. Compounding error destroys this calibration.

In model-based offline RL, where the agent cannot interact with the real environment, compounding error is a primary failure mode. The policy is trained solely on synthetic rollouts from a model learned on static data. Any error compounds without the possibility of correction, often leading the policy to exploit extrapolation errors in the model and propose actions far outside the training data distribution, with unpredictable results.

• Critical Concern: This makes deploying MBRL agents trained offline in safety-critical domains (e.g., healthcare, finance) exceptionally risky without rigorous sim-to-real validation and safeguards.

The planning horizon is the number of future time steps an agent considers when simulating trajectories. It is a critical trade-off parameter. A longer horizon allows for more strategic, long-term decisions but exponentially increases the impact of compounding error as inaccuracies accumulate. Algorithms like Model Predictive Control (MPC) use a receding horizon, executing only the first planned action before re-planning, to limit exposure to distant, error-prone predictions.

Model-policy co-adaptation is a pernicious failure mode where a policy learns to exploit the specific biases and inaccuracies of its own learned dynamics model. This creates a feedback loop: the policy drives the agent into regions of state space where the model is over-optimistically accurate, but which are unrealistic in the true environment. This pathological synergy can mask compounding error during training, leading to catastrophic performance collapse upon deployment.

Uncertainty quantification involves estimating the confidence of a dynamics model's predictions. It is the primary technical defense against compounding error. Methods include:

• Probabilistic Ensembles: Using multiple models; disagreement indicates epistemic uncertainty.
• Bayesian Neural Networks (BNNs): Representing weights as distributions to capture uncertainty. Planners can use this uncertainty to avoid states where predictions are unreliable (pessimistic exploration) or to seek them out for improved learning (model-based exploration).

Certainty-equivalence control is a naive planning strategy where an agent acts as if its learned dynamics model is perfectly accurate, completely ignoring predictive uncertainty. This approach is highly susceptible to compounding error, as even small inaccuracies are treated as truth and propagated forward. It often leads to catastrophic failures when the agent encounters states outside its model's reliable domain, highlighting why robust MBRL requires explicit uncertainty handling.