Model-Policy Co-adaptation

The primary driver of co-adaptation is the compounding error inherent in multi-step rollouts. A small inaccuracy in the learned transition model is amplified each time the model predicts a subsequent state. The policy, trained via planning or model-based policy optimization (MBPO) on these erroneous trajectories, learns to exploit these inaccuracies as if they were real environment dynamics. This creates a feedback loop where the policy's behavior reinforces the model's biases.

Co-adaptation occurs when planning algorithms, like Model Predictive Control (MPC) or trajectory optimization, treat the learned model's predictions with certainty-equivalence. They ignore epistemic uncertainty (model uncertainty). Without uncertainty quantification from techniques like Bayesian Neural Networks (BNNs) or probabilistic ensembles, the policy is optimized for a single, potentially wrong, view of the world. It learns actions that are optimal only for this flawed internal simulation.

The policy is trained on a distribution of states generated by the model's imagined rollouts. As the policy adapts to exploit model errors, it visits regions of the state space in simulation that are improbable or impossible in the real environment. This creates a distributional shift between the training data (simulated states) and test data (real states). The policy becomes highly specialized to this synthetic distribution and fails to generalize.

• Example: A robot arm policy learns to make ultra-fast movements that are physically plausible in a low-fidelity physics simulator but cause instability or damage on real hardware.

The policy acts as a powerful search algorithm, finding trajectories that maximize predicted reward according to the model. If the model has systematic biases—for example, underestimating friction or overestimating object durability—the policy will relentlessly exploit these biases. It learns a degenerate solution that scores highly in simulation but is ineffective or dangerous in reality. This is distinct from model error; it is the active optimization of policy behavior to align with those errors.

In pure model-based training paradigms like Dreamer, the policy is trained exclusively on latent imagination within a world model (e.g., a Recurrent State-Space Model). Without periodic validation and regularization against ground-truth environment interactions, there is no mechanism to correct the policy's drift away from reality. The policy and model enter a closed co-adaptation loop, each adapting to the other's peculiarities, further decoupling from the true system identification target.

Successful algorithms prevent co-adaptation by introducing regularization that grounds the policy in reality.

• Pessimistic Exploration: Penalizes actions in states where the model is uncertain (common in model-based offline RL).
• Limited Planning Horizons: Using short planning horizons in MPC reduces the impact of compounding error.
• Hybrid Training: Algorithms like MBPO mix limited real experience with model-generated data to prevent distributional drift.
• Value-Equivalent Models: As in MuZero, learning a model that is only accurate for planning purposes can be more robust than learning a perfect dynamics model.

Compounding error is the phenomenon where small inaccuracies in a learned dynamics model accumulate multiplicatively over the course of a multi-step imagined rollout. This leads to simulated states that diverge exponentially from what would occur in the real environment.

• Primary Cause: Imperfect model predictions at each timestep.
• Consequence: Policies trained on these unrealistic rollouts learn to exploit the model's flawed reality, directly enabling model-policy co-adaptation.
• Mitigation: Techniques include using short planning horizons, uncertainty-aware planning, and regularizing the policy against exploiting model errors.

Model error is the fundamental discrepancy between the predictions of a learned transition model (or dynamics model) and the true environment dynamics. It is the root source of performance degradation in MBRL.

• Types: Can be epistemic (due to lack of data) or aleatoric (due to inherent environment stochasticity).
• Role in Co-adaptation: A policy that overfits to a specific pattern of model error, rather than learning robust behaviors, is the essence of co-adaptation. Managing model error through uncertainty quantification is key to prevention.

Uncertainty quantification involves estimating the confidence of a learned model's predictions. It is a critical defense against model-policy co-adaptation, as it allows an agent to know when its model is unreliable.

• Methods: Common approaches include Bayesian Neural Networks (BNNs) and probabilistic ensembles, where disagreement among ensemble members indicates high uncertainty.
• Application: Used in pessimistic exploration and robust planning algorithms (e.g., planning with upper confidence bounds) to avoid exploiting states where the model is likely wrong, thereby preventing policy overfitting to model biases.

Pessimistic exploration (or conservative model-based RL) is a strategy where an agent's policy is explicitly constrained or penalized to avoid states and actions where the learned dynamics model has high uncertainty. This is crucial for model-based offline RL and preventing co-adaptation.

• Mechanism: The agent assumes the worst-case outcome in uncertain regions, preventing it from exploiting optimistic but inaccurate model predictions.
• Benefit: Drastically improves robustness and safety by ensuring the policy remains within the support of the data used to train the model, mitigating the risk of co-adaptation to model fantasies.

Model-Based Policy Optimization (MBPO) is a prominent algorithm that exemplifies the tension leading to co-adaptation. It uses short, imagined rollouts from a learned model to generate synthetic data for training a policy via model-free methods like SAC.

• Process: The policy is updated using data from the model, creating a tight feedback loop.
• Risk: If the model rollouts are biased, the policy quickly adapts to perform well on those biased rollouts, a direct instance of model-policy co-adaptation. Successful MBPO implementations carefully manage rollout horizon and incorporate model uncertainty to avoid this pitfall.

A world model is an agent's internal, learned representation that predicts future states and rewards. It is the core component in algorithms like Dreamer that enables imagined rollouts.

• Relation to Co-adaptation: In advanced architectures like Recurrent State-Space Models (RSSM), the policy is trained entirely through latent imagination—backpropagation through time on sequences generated by the world model. This creates a perfect environment for co-adaptation if the world model's latent dynamics are inaccurate. The policy and world model can become a closed, self-reinforcing system detached from reality.