Model-Policy Co-adaptation

The policy is trained on a distribution of states generated by the model's imagined rollouts. As the policy adapts to exploit model errors, it visits regions of the state space in simulation that are improbable or impossible in the real environment. This creates a distributional shift between the training data (simulated states) and test data (real states). The policy becomes highly specialized to this synthetic distribution and fails to generalize.

• Example: A robot arm policy learns to make ultra-fast movements that are physically plausible in a low-fidelity physics simulator but cause instability or damage on real hardware.

The policy acts as a powerful search algorithm, finding trajectories that maximize predicted reward according to the model. If the model has systematic biases—for example, underestimating friction or overestimating object durability—the policy will relentlessly exploit these biases. It learns a degenerate solution that scores highly in simulation but is ineffective or dangerous in reality. This is distinct from model error; it is the active optimization of policy behavior to align with those errors.

In pure model-based training paradigms like Dreamer, the policy is trained exclusively on latent imagination within a world model (e.g., a Recurrent State-Space Model). Without periodic validation and regularization against ground-truth environment interactions, there is no mechanism to correct the policy's drift away from reality. The policy and model enter a closed co-adaptation loop, each adapting to the other's peculiarities, further decoupling from the true system identification target.

Successful algorithms prevent co-adaptation by introducing regularization that grounds the policy in reality.

• Pessimistic Exploration: Penalizes actions in states where the model is uncertain (common in model-based offline RL).
• Limited Planning Horizons: Using short planning horizons in MPC reduces the impact of compounding error.
• Hybrid Training: Algorithms like MBPO mix limited real experience with model-generated data to prevent distributional drift.
• Value-Equivalent Models: As in MuZero, learning a model that is only accurate for planning purposes can be more robust than learning a perfect dynamics model.

Uncertainty quantification involves estimating the confidence of a learned model's predictions. It is a critical defense against model-policy co-adaptation, as it allows an agent to know when its model is unreliable.

• Methods: Common approaches include Bayesian Neural Networks (BNNs) and probabilistic ensembles, where disagreement among ensemble members indicates high uncertainty.
• Application: Used in pessimistic exploration and robust planning algorithms (e.g., planning with upper confidence bounds) to avoid exploiting states where the model is likely wrong, thereby preventing policy overfitting to model biases.

Pessimistic exploration (or conservative model-based RL) is a strategy where an agent's policy is explicitly constrained or penalized to avoid states and actions where the learned dynamics model has high uncertainty. This is crucial for model-based offline RL and preventing co-adaptation.

• Mechanism: The agent assumes the worst-case outcome in uncertain regions, preventing it from exploiting optimistic but inaccurate model predictions.
• Benefit: Drastically improves robustness and safety by ensuring the policy remains within the support of the data used to train the model, mitigating the risk of co-adaptation to model fantasies.

Model-Based Policy Optimization (MBPO) is a prominent algorithm that exemplifies the tension leading to co-adaptation. It uses short, imagined rollouts from a learned model to generate synthetic data for training a policy via model-free methods like SAC.

• Process: The policy is updated using data from the model, creating a tight feedback loop.
• Risk: If the model rollouts are biased, the policy quickly adapts to perform well on those biased rollouts, a direct instance of model-policy co-adaptation. Successful MBPO implementations carefully manage rollout horizon and incorporate model uncertainty to avoid this pitfall.

A world model is an agent's internal, learned representation that predicts future states and rewards. It is the core component in algorithms like Dreamer that enables imagined rollouts.

• Relation to Co-adaptation: In advanced architectures like Recurrent State-Space Models (RSSM), the policy is trained entirely through latent imagination—backpropagation through time on sequences generated by the world model. This creates a perfect environment for co-adaptation if the world model's latent dynamics are inaccurate. The policy and world model can become a closed, self-reinforcing system detached from reality.