Certainty-Equivalence Control

The agent acts as if its learned dynamics model is perfectly accurate. It treats the model's predictions as ground truth, ignoring any predictive uncertainty or model error. This assumption simplifies planning to a deterministic optimization problem but is the root cause of its primary weakness: catastrophic failure when the model is wrong.

By ignoring uncertainty, the planning problem reduces to finding an optimal action sequence for a deterministic transition function. This allows the use of fast, classical trajectory optimization techniques like:

• Iterative Linear Quadratic Regulator (iLQR)
• Model Predictive Control (MPC) with deterministic rollouts
• Gradient-based shooting methods This makes it computationally attractive for real-time control in well-modeled systems.

This is the most critical failure mode. Small inaccuracies in the dynamics model are not just ignored; they accumulate multiplicatively over the course of an imagined rollout. A state predicted 10 steps into the future may bear little resemblance to reality, causing the agent to plan optimal actions for a fictional world. This often leads to catastrophic failures or irrecoverable states upon execution.

Certainty-equivalence is the antithesis of modern robust MBRL approaches:

• Robust Control: Plans for a set of possible dynamics (worst-case).
• Pessimistic Exploration: Penalizes actions in high-uncertainty states.
• Probabilistic Ensembles: Uses multiple models to estimate and account for uncertainty. Certainty-equivalence provides none of these safety guarantees, making it unsuitable for safety-critical or poorly understood environments.

Despite its risks, certainty-equivalence can be effective in specific, constrained scenarios:

• High-Fidelity Simulators: Where the model is effectively perfect (e.g., classic robotics with accurate physics engines).
• System Identification: After extensive data collection has minimized model error.
• Short Planning Horizons: Where compounding error has minimal time to accumulate.
• Baseline Algorithm: Serves as a simple benchmark against which more sophisticated, uncertainty-aware methods are compared.

Certainty-equivalence control is highly prone to model-policy co-adaptation, a degenerative failure mode. The policy learns to exploit the specific biases and inaccuracies of its own learned model, creating a synergistic failure. The policy performs well in simulations using the flawed model but fails catastrophically in the real environment, as the policy and model have co-evolved to be optimal for each other, not for reality.

Model Predictive Control (MPC) is an online, receding-horizon planning algorithm that repeatedly solves a finite-horizon optimal control problem using a learned dynamics model. Unlike certainty-equivalence, which commits to a single open-loop plan, MPC executes only the first action from the optimized sequence before replanning from the new observed state. This feedback mechanism makes it more robust to small model errors and environmental disturbances.

• Key Contrast: MPC inherently incorporates feedback, while certainty-equivalence is purely open-loop.
• Computational Cost: MPC is more computationally intensive due to frequent re-optimization.
• Use Case: Dominant in robotics and process control where real-time sensor feedback is available.

Pessimistic exploration, or conservative model-based RL, is a planning philosophy directly opposed to certainty-equivalence. It explicitly accounts for model uncertainty to avoid catastrophic failures. The agent's policy is constrained or penalized to avoid states and actions where the learned dynamics model is highly uncertain.

• Core Mechanism: Uses uncertainty estimates (e.g., from a probabilistic ensemble) to shape a conservative value function or policy.
• Primary Application: Critical for model-based offline RL, where the agent cannot interact with the real environment to correct model mistakes.
• Result: Sacrifices some potential performance for greatly improved robustness and safety.

Trajectory optimization is a broad class of planning methods that search for a sequence of actions minimizing a cost function (or maximizing reward) over a horizon, subject to a dynamics model. Certainty-equivalence is a specific, simplistic instance of trajectory optimization that ignores uncertainty.

• Advanced Methods: Include the Iterative Linear Quadratic Regulator (iLQR), which iteratively linearizes dynamics and quadratizes cost for efficient optimization.
• Input: Requires a differentiable dynamics model and cost function.
• Output: Produces an optimal open-loop action sequence and associated state trajectory, which certainty-equivalence executes naively.

Model error is the discrepancy between a learned dynamics model's predictions and the true environment. Compounding error is the critical failure mode that makes certainty-equivalence control dangerous: small inaccuracies in the model's one-step predictions accumulate exponentially over the course of a multi-step imagined rollout.

• Consequence: The agent's planned trajectory rapidly diverges from reality, leading to actions that are optimal in simulation but catastrophic in the real world.
• Mitigation: Techniques like short planning horizons, model ensembles, and replanning (MPC) are used to bound this error.
• Certainty-Equivalence Assumption: Effectively assumes both model error and compounding error are zero.

System identification is the classical field of learning a mathematical model of a system's dynamics from observed input-output data. It is the foundational step that provides the dynamics model used in certainty-equivalence control and other MBRL methods.

• Scope: Encompasses both simple linear regression and complex deep learning for transition models.
• Goal: To minimize model error on the training distribution.
• Limitation for CE Control: Traditional system identification focuses on average prediction accuracy, not the uncertainty quantification needed to assess the risk of using the model for long-horizon, open-loop control.

Model-policy co-adaptation is a subtle failure mode relevant to iterative MBRL algorithms, highlighting a risk beyond simple certainty-equivalence. It occurs when a policy is trained extensively (e.g., via model-based policy optimization) on synthetic data from its own learned model.

• Process: The policy learns to exploit the specific biases and inaccuracies of its own model, achieving high reward in simulation.
• Result: This creates a distributional shift where the policy performs poorly when deployed in the real environment, as it is overfitted to the model's errors.
• Contrast to CE: CE is a one-shot planning failure; co-adaptation is a training-time failure from repeated interaction between a flawed model and a policy.