Model Predictive Control (MPC)

MPC does not use a fixed policy function. Instead, it solves a numerical optimization problem at every step. This problem minimizes (or maximizes) a defined cost function J over the planning horizon H:

J = Σ_{k=0}^{H-1} cost(state_k, action_k) + terminal_cost(state_H)

The cost function encodes the control objective, such as tracking a reference trajectory, minimizing energy use, or maximizing reward. This allows for multi-objective tuning by weighting different cost terms.

MPC relies on a predictive model of the system dynamics to simulate future states. In classical control, this is often an analytical model (e.g., differential equations). In Model-Based RL, this is a learned dynamics model (e.g., a neural network). The quality of this model is paramount:

• Transition Model: Predicts the next state: s_{t+1} = f_θ(s_t, a_t).
• Reward/Cost Model: Predicts the immediate cost/reward.
• Model Error: Inaccuracy in f_θ leads to compounding error over long rollouts, a key challenge in MBRL.

The planning horizon H is a critical hyperparameter that creates a fundamental trade-off:

• Long Horizon: Enables better long-term decision-making, avoiding myopic policies. Essential for tasks with delayed rewards or complex maneuvers.
• Short Horizon: Reduces computational cost per planning step and mitigates the impact of compounding model error.

In practice, H is chosen to be long enough to capture the system's relevant dynamics but short enough to allow for real-time optimization (often < 50 steps).

MPC is fundamentally an online algorithm. The optimization is performed in real-time during deployment. This contrasts with offline policy learning (e.g., in model-free RL), where a policy is trained once and then executed with minimal computation.

• Pro: Adapts to novel situations not seen in training.
• Con: Requires significant and reliable computational resources at inference time.
• Solvers: Efficient numerical solvers (e.g., for quadratic programs, differential dynamic programming) are essential. In learned settings, algorithms like MuZero use Monte Carlo Tree Search as the online planner.

Trajectory optimization is the mathematical core of the MPC planning step. It searches for a sequence of actions that minimizes a cost (or maximizes reward) over a finite planning horizon, subject to the constraints defined by the dynamics model. MPC repeatedly solves this optimization problem online.

• Methods: Includes both gradient-based techniques (e.g., Iterative Linear Quadratic Regulator (iLQR)) and sampling-based methods.
• Input: Current state and a dynamics/reward model.
• Output: An optimal (or near-optimal) action sequence, of which only the first action is executed.

The planning horizon is the number of future time steps an MPC algorithm looks ahead during each trajectory optimization cycle. It is a critical hyperparameter that balances decision quality against computational cost.

• Short Horizon: Computationally cheap but myopic; may miss long-term rewards or avoid short-term costs leading to long-term gain.
• Long Horizon: Enables foresight but is computationally expensive and more susceptible to compounding error from an imperfect model.
• Tuning: Often set based on the problem's time constants and available compute budget for real-time control.

Certainty-equivalence control is a naive planning strategy where an agent acts as if its learned model's predictions are perfectly accurate, completely ignoring predictive uncertainty. This is a common, simple baseline within MPC that can fail catastrophically if the model is wrong.

• Contrast with Robust MPC: Advanced MPC variants explicitly incorporate uncertainty quantification (e.g., from Bayesian Neural Networks or probabilistic ensembles) to plan conservatively.
• Risk: Leads to model-policy co-adaptation, where the policy exploits model flaws, performing poorly in the real world.
• Example: Using a single, deterministic neural network as a dynamics model without any uncertainty-aware planning.

Sample efficiency measures the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the principal motivation for using model-based methods like MPC.

• Model-Free RL: Often requires millions to billions of environment steps.
• Model-Based RL/MPC: Aims to learn a usable model with thousands to hundreds of thousands of steps, then uses planning to achieve good performance.
• Trade-off: The computational cost of planning is traded for reduced real-world data collection, which is crucial when interactions are expensive, risky, or slow (e.g., robotics, autonomous vehicles).