Compounding Error

The most direct consequence is that an agent's planned trajectory in its internal model deviates exponentially from what is physically possible in the real environment. A small error in predicting state s_t+1 becomes a massive error at s_t+10. This renders long-horizon planning useless, as the agent optimizes for futures that cannot occur.

• Example: A robot arm planning a 10-step manipulation sequence may believe an object is within grasp by step 10, while in reality, a 1cm positional error at step 2 has compounded, placing the object completely out of reach.

Policies can co-adapt with their own flawed dynamics model, learning to exploit its inaccuracies to achieve artificially high simulated rewards. This is a pathological form of overfitting where the policy performs well in the model but fails catastrophically in the real world. The agent finds 'shortcuts' in the simulation that don't exist.

• Mechanism: The policy gradient update is computed using imagined states. If the model consistently underestimates friction, the policy may learn to apply insufficient force, causing real-world tasks to fail.

The core promise of MBRL—high sample efficiency—is negated. If rollouts are too short to avoid compounding error, little useful synthetic data is generated. If they are too long, the synthetic data is corrupted and poisons policy training. Engineers must then fall back to costly real-environment interaction to correct the policy, erasing MBRL's primary advantage.

• Quantitative Impact: An algorithm like Model-Based Policy Optimization (MBPO) relies on short, accurate rollouts. Compounding error forces shorter horizons, reducing the value of each imagined rollout and requiring more real data collection.

Model Predictive Control (MPC), which replans at each step, is particularly vulnerable. While replanning mitigates error by correcting course, a severely inaccurate model means every new plan starts from a flawed belief state and compounds anew. The agent is perpetually 'chasing its tail,' leading to hesitant, oscillatory, or unstable behavior in the real environment.

• Real-World Effect: An autonomous vehicle using MPC with a poor dynamics model may exhibit jerky, over-corrective steering as each new plan based on faulty predictions leads to another unexpected state.

Compounding error corrupts uncertainty quantification. An agent cannot distinguish between states that are truly uncertain/novel and states that are simply miscalculated. This undermines pessimistic exploration strategies designed for safety. The agent may avoid safe, known states (due to imagined error) or confidently enter dangerous ones (due to unrealistically certain predictions).

• Link to Uncertainty: Methods using Probabilistic Ensembles or Bayesian Neural Networks to estimate uncertainty rely on the model's ability to self-assess. Compounding error destroys this calibration.

In model-based offline RL, where the agent cannot interact with the real environment, compounding error is a primary failure mode. The policy is trained solely on synthetic rollouts from a model learned on static data. Any error compounds without the possibility of correction, often leading the policy to exploit extrapolation errors in the model and propose actions far outside the training data distribution, with unpredictable results.

• Critical Concern: This makes deploying MBRL agents trained offline in safety-critical domains (e.g., healthcare, finance) exceptionally risky without rigorous sim-to-real validation and safeguards.

Model error is the fundamental discrepancy between the predictions of a learned dynamics model and the true environment dynamics. It is the primary source from which compounding error originates. This error can be decomposed into:

• Epistemic uncertainty: Inaccuracy due to insufficient training data or model capacity.
• Aleatoric uncertainty: Inherent stochasticity in the environment that is impossible to predict perfectly. Managing model error through robust architectures and uncertainty quantification is essential to mitigate compounding error.

A world model is an agent's internal, learned representation used to simulate future states and rewards. It is the engine for imagined rollouts, where compounding error manifests. High-fidelity world models, such as Recurrent State-Space Models (RSSM), aim to learn compressed latent representations to improve generalization and reduce prediction drift over long horizons. The accuracy of this model directly dictates the severity of compounding error during multi-step planning.

The planning horizon is the number of future time steps an agent considers when simulating trajectories. It is a critical trade-off parameter. A longer horizon allows for more strategic, long-term decisions but exponentially increases the impact of compounding error as inaccuracies accumulate. Algorithms like Model Predictive Control (MPC) use a receding horizon, executing only the first planned action before re-planning, to limit exposure to distant, error-prone predictions.

Model-policy co-adaptation is a pernicious failure mode where a policy learns to exploit the specific biases and inaccuracies of its own learned dynamics model. This creates a feedback loop: the policy drives the agent into regions of state space where the model is over-optimistically accurate, but which are unrealistic in the true environment. This pathological synergy can mask compounding error during training, leading to catastrophic performance collapse upon deployment.

Uncertainty quantification involves estimating the confidence of a dynamics model's predictions. It is the primary technical defense against compounding error. Methods include:

• Probabilistic Ensembles: Using multiple models; disagreement indicates epistemic uncertainty.
• Bayesian Neural Networks (BNNs): Representing weights as distributions to capture uncertainty. Planners can use this uncertainty to avoid states where predictions are unreliable (pessimistic exploration) or to seek them out for improved learning (model-based exploration).

Certainty-equivalence control is a naive planning strategy where an agent acts as if its learned dynamics model is perfectly accurate, completely ignoring predictive uncertainty. This approach is highly susceptible to compounding error, as even small inaccuracies are treated as truth and propagated forward. It often leads to catastrophic failures when the agent encounters states outside its model's reliable domain, highlighting why robust MBRL requires explicit uncertainty handling.