Trajectory Optimization

Trajectory optimization solves for an optimal sequence of actions over a defined, finite number of future time steps, known as the planning horizon. This contrasts with infinite-horizon methods common in policy optimization. The horizon length is a critical trade-off: a longer horizon enables better long-term planning but increases computational cost and susceptibility to model error.

The method is fundamentally reliant on a dynamics model (or transition model) that predicts the next state given the current state and action. This model can be:

• Analytical: Derived from first principles (e.g., physics equations for a robot arm).
• Learned: A neural network trained on interaction data, as in Model-Based Reinforcement Learning (MBRL). Planning occurs within this internal simulation, enabling sample-efficient evaluation of candidate action sequences without real-world trial-and-error.

The core objective is to find the action trajectory that minimizes a scalar cost function (or equivalently, maximizes cumulative reward). This function encodes the task goals, such as:

• Reaching a target state with minimal error.
• Minimizing control effort or energy consumption.
• Avoiding obstacles or unsafe states via penalty terms. The optimizer's job is to navigate the high-dimensional space of possible trajectories to find the one with the lowest total cost.

Efficient solvers leverage gradient information from the dynamics and cost models. The most prominent algorithm is the Iterative Linear Quadratic Regulator (iLQR) and its stochastic variant, iLQG. These methods:

1. Iteratively linearize the dynamics around a current trajectory guess.
2. Quadratize the cost function.
3. Solve the resulting LQR problem efficiently via dynamic programming.
4. Update the trajectory and repeat. This provides fast convergence to a locally optimal solution.

In practice, trajectory optimization is often deployed within a Model Predictive Control (MPC) loop. At each control cycle:

1. The current state is observed.
2. A new optimal trajectory is computed from this state.
3. Only the first action of the planned sequence is executed.
4. The process repeats at the next time step. This receding horizon control provides robustness to model inaccuracies and unexpected disturbances.

Trajectory optimization is a planning method, distinct from policy search in reinforcement learning. Key differences:

• Output: Trajectory optimization outputs a specific action sequence for a given start state. Policy search learns a function (policy) mapping any state to an action.
• Online Computation: Planning is computationally intensive at runtime. A trained policy offers cheap, constant-time action selection.
• Use Case: Planning is ideal for problems with accurate models and where conditions vary (e.g., robot arm reaching for different objects). Policies are better for fast reaction in fixed environments.

Model Predictive Control (MPC) is an online, receding-horizon control strategy that uses a dynamics model for planning. At each time step, MPC solves a finite-horizon trajectory optimization problem from the current state, executes only the first planned action, and then replans from the new observed state. This feedback loop makes it robust to model inaccuracies and environmental disturbances. It is widely used in robotics and process control.

• Core Mechanism: Replanning at every step based on new observations.
• Key Benefit: Inherent robustness to model error through frequent re-optimization.
• Common Use Case: Real-time control of autonomous vehicles and robotic manipulators.

The Iterative Linear Quadratic Regulator (iLQR) is a specific, highly efficient trajectory optimization algorithm. It operates by iteratively linearizing the system dynamics around a current trajectory and quadratizing the cost function. This transforms the complex nonlinear problem into a series of Linear Quadratic Regulator (LQR) problems, which can be solved exactly and efficiently via dynamic programming. The algorithm converges quickly to a locally optimal trajectory.

• Core Mechanism: Iterative local approximation (linearize dynamics, quadratize cost).
• Key Benefit: Second-order convergence speed for smooth systems.
• Prerequisite: Requires differentiable dynamics and cost models.

The planning horizon is the number of future time steps (denoted as H) that an agent considers when simulating trajectories during optimization. It is a critical hyperparameter that balances computational cost against decision quality.

• Short Horizon: Computationally cheap but can lead to myopic, suboptimal policies (e.g., avoiding a small immediate cost that leads to a large future reward).
• Long Horizon: Enables long-term strategic planning but is exponentially more expensive and more susceptible to compounding error from an imperfect model.
• Trade-off: Optimal horizon length depends on task complexity, model accuracy, and available compute.

Compounding error is a fundamental challenge in model-based RL where inaccuracies in a learned dynamics model accumulate multiplicatively over the course of a multi-step imagined rollout. A small error in predicting the next state leads to the model being applied in a state it has never seen before, causing subsequent predictions to diverge further from reality.

• Consequence: Long-horizon plans generated using the model can lead to completely unrealistic simulated states, causing the optimized policy to fail in the real environment.
• Mitigation Strategies: Using shorter planning horizons (MPC), employing uncertainty quantification to avoid uncertain states, and algorithms designed for robustness to model error.

Certainty-equivalence control is a naive planning approach where an agent acts as if its learned dynamics model is a perfect, deterministic representation of the true environment. The agent ignores all predictive uncertainty and solves the trajectory optimization problem under the assumption the model is correct.

• Risk: This approach is highly susceptible to catastrophic failure when the model is inaccurate, as the agent may confidently plan trajectories through physically impossible or dangerous state regions.
• Contrast: Advanced MBRL methods explicitly account for model uncertainty, using techniques like probabilistic ensembles or Bayesian Neural Networks (BNNs) to guide pessimistic exploration or robust planning.

System identification is the classical field of building mathematical models of dynamic systems from observed input-output data. In the context of MBRL, it is the process of learning a transition model (dynamics) and often a reward model from interaction data. This learned model is the foundational component used for subsequent trajectory optimization.

• Methods: Range from linear regression for simple systems to deep neural networks for complex, high-dimensional environments (e.g., pixels).
• Goal: To obtain a model that is accurate enough for effective planning, not necessarily a perfect replica of the world.
• Connection: MBRL can be viewed as integrating system identification with optimal control.