Model Predictive Control (MPC)

This is the defining mechanism of MPC. At each control step, the algorithm solves a finite-horizon optimal control problem but executes only the first action from the computed optimal sequence. It then shifts the planning window forward by one time step, observes the new state, and replans. This creates a feedback loop that continuously corrects for model errors and external disturbances.

• Real-time Adaptation: Continuously incorporates the latest observations.
• Inherent Robustness: Mitigates the impact of disturbances and modeling inaccuracies by frequent re-optimization.

A major advantage of MPC over other control methods is its ability to directly incorporate hard and soft constraints into the online optimization problem. These can include:

• State Constraints: e.g., joint limits for a robot, safe operating temperatures.
• Input Constraints: e.g., actuator torque/speed limits, voltage boundaries.
• Output Constraints: e.g., keeping a vehicle within lane boundaries.

The optimizer finds a control sequence that satisfies these constraints over the planning horizon, ensuring safe and feasible operation.

MPC does not use a fixed policy function. Instead, it solves a numerical optimization problem at every step. This problem minimizes (or maximizes) a defined cost function J over the planning horizon H:

J = Σ_{k=0}^{H-1} cost(state_k, action_k) + terminal_cost(state_H)

The cost function encodes the control objective, such as tracking a reference trajectory, minimizing energy use, or maximizing reward. This allows for multi-objective tuning by weighting different cost terms.

MPC relies on a predictive model of the system dynamics to simulate future states. In classical control, this is often an analytical model (e.g., differential equations). In Model-Based RL, this is a learned dynamics model (e.g., a neural network). The quality of this model is paramount:

• Transition Model: Predicts the next state: s_{t+1} = f_θ(s_t, a_t).
• Reward/Cost Model: Predicts the immediate cost/reward.
• Model Error: Inaccuracy in f_θ leads to compounding error over long rollouts, a key challenge in MBRL.

The planning horizon H is a critical hyperparameter that creates a fundamental trade-off:

• Long Horizon: Enables better long-term decision-making, avoiding myopic policies. Essential for tasks with delayed rewards or complex maneuvers.
• Short Horizon: Reduces computational cost per planning step and mitigates the impact of compounding model error.

In practice, H is chosen to be long enough to capture the system's relevant dynamics but short enough to allow for real-time optimization (often < 50 steps).

MPC is fundamentally an online algorithm. The optimization is performed in real-time during deployment. This contrasts with offline policy learning (e.g., in model-free RL), where a policy is trained once and then executed with minimal computation.

• Pro: Adapts to novel situations not seen in training.
• Con: Requires significant and reliable computational resources at inference time.
• Solvers: Efficient numerical solvers (e.g., for quadratic programs, differential dynamic programming) are essential. In learned settings, algorithms like MuZero use Monte Carlo Tree Search as the online planner.

Model-Based Reinforcement Learning (MBRL) is the overarching paradigm where an agent learns an internal dynamics model and reward model of its environment. This model is then used for planning and policy optimization, with the primary goal of improving sample efficiency compared to model-free methods. MPC is a specific, online planning algorithm employed within MBRL systems.

• Core Idea: Replace millions of real-world trials with internal simulation.
• Key Challenge: Managing model error to prevent compounding error during long rollouts.
• Example Algorithms: Dreamer, MuZero, and Model-Based Policy Optimization (MBPO).

A world model is the learned internal representation that allows an agent to predict future states and rewards. It serves as a simulated environment for planning algorithms like MPC. Crucially, a world model operates in a potentially compressed latent space, especially for high-dimensional observations like images.

• Function: Enables "imagination" or imagined rollouts without real interaction.
• Architecture: Often implemented as a Recurrent State-Space Model (RSSM) to handle partial observability and temporal dependencies.
• Utility: The accuracy and generality of the world model directly limit the effectiveness of MPC.

Trajectory optimization is the mathematical core of the MPC planning step. It searches for a sequence of actions that minimizes a cost (or maximizes reward) over a finite planning horizon, subject to the constraints defined by the dynamics model. MPC repeatedly solves this optimization problem online.

• Methods: Includes both gradient-based techniques (e.g., Iterative Linear Quadratic Regulator (iLQR)) and sampling-based methods.
• Input: Current state and a dynamics/reward model.
• Output: An optimal (or near-optimal) action sequence, of which only the first action is executed.

The planning horizon is the number of future time steps an MPC algorithm looks ahead during each trajectory optimization cycle. It is a critical hyperparameter that balances decision quality against computational cost.

• Short Horizon: Computationally cheap but myopic; may miss long-term rewards or avoid short-term costs leading to long-term gain.
• Long Horizon: Enables foresight but is computationally expensive and more susceptible to compounding error from an imperfect model.
• Tuning: Often set based on the problem's time constants and available compute budget for real-time control.

Certainty-equivalence control is a naive planning strategy where an agent acts as if its learned model's predictions are perfectly accurate, completely ignoring predictive uncertainty. This is a common, simple baseline within MPC that can fail catastrophically if the model is wrong.

• Contrast with Robust MPC: Advanced MPC variants explicitly incorporate uncertainty quantification (e.g., from Bayesian Neural Networks or probabilistic ensembles) to plan conservatively.
• Risk: Leads to model-policy co-adaptation, where the policy exploits model flaws, performing poorly in the real world.
• Example: Using a single, deterministic neural network as a dynamics model without any uncertainty-aware planning.

Sample efficiency measures the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the principal motivation for using model-based methods like MPC.

• Model-Free RL: Often requires millions to billions of environment steps.
• Model-Based RL/MPC: Aims to learn a usable model with thousands to hundreds of thousands of steps, then uses planning to achieve good performance.
• Trade-off: The computational cost of planning is traded for reduced real-world data collection, which is crucial when interactions are expensive, risky, or slow (e.g., robotics, autonomous vehicles).