Planning Horizon

The optimal planning horizon is intrinsically linked to the fidelity of the agent's learned world model.

• High-Fidelity Model: Can support longer horizons, as predictions remain reliable deep into the future. Algorithms like Dreamer leverage accurate latent dynamics models for long-horizon imagination.
• Noisy or Inaccurate Model: Necessitates a shorter, more conservative horizon to prevent the policy from exploiting model hallucinations. Techniques like pessimistic exploration or uncertainty quantification (e.g., using a probabilistic ensemble) can inform adaptive horizon selection.

The planning horizon is implemented differently across key MBRL algorithms:

• Model Predictive Control (MPC): Uses a fixed, receding horizon. At each step, it plans an optimal trajectory over H steps, executes the first action, then re-plans.
• Monte Carlo Tree Search (MCTS): The horizon defines the maximum depth of the search tree before a rollout or evaluation function is called.
• Trajectory Optimization (e.g., iLQR): The horizon sets the length of the control sequence to be optimized.
• Policy Learning via Imagination (e.g., MBPO): The horizon determines the length of imagined rollouts used to generate synthetic training data for the policy.

Advanced strategies move beyond a fixed horizon.

• Adaptive Horizons: The horizon can be dynamically adjusted based on model uncertainty or task complexity, shortening in uncertain regions and extending where the model is confident.
• Discounting and Effective Horizons: In infinite-horizon problems, a discount factor (γ) creates an effective horizon. The agent's planning is weighted towards near-term futures, as rewards far in the future are geometrically discounted. The planning horizon in simulation is often truncated at a point where further discounted rewards become negligible.

Setting the planning horizon is a key hyperparameter tuning task.

• Start Short: Begin with a horizon just long enough to solve simple sub-tasks, then increase incrementally while monitoring real-world performance degradation from model-policy co-adaptation.
• Benchmark Against Model-Free: Compare the sample efficiency and final performance of your MBRL agent with varying horizons against a strong model-free RL baseline to validate the benefit of planning.
• Hardware Constraints: The maximum feasible horizon is often dictated by available compute and latency requirements for real-time systems (e.g., robotics).

Compounding Error is a fundamental challenge in model-based RL where small inaccuracies in a learned transition model multiply over the course of a long imagined rollout. The predicted state diverges exponentially from the true state the environment would produce.

• Primary Cause: Imperfect model learning from finite data.
• Impact on Planning: Renders long-horizon predictions unreliable, forcing a practical limit on the useful planning horizon.
• Mitigation Strategies:
  • Using probabilistic ensembles to quantify and penalize uncertain predictions.
  • Implementing pessimistic exploration to avoid exploiting model errors.
  • Employing shorter rollouts for policy training (as in MBPO).

Imagined Rollouts are sequences of states, actions, and rewards generated by sampling actions and unrolling a learned world model forward in time. They provide cheap, synthetic training data for the policy and value functions.

• Purpose: Enables sample-efficient learning by reducing real-environment interactions.
• Architectures: Central to algorithms like Dreamer (latent imagination) and MBPO (short-horizon model rollouts).
• Horizon Length: A key design choice. Short rollouts (e.g., 1-5 steps in MBPO) reduce compounding error, while longer rollouts in latent space (Dreamer) allow for more coherent long-term planning.

Certainty-Equivalence Control is a naive planning approach where an agent acts as if its learned dynamics model is perfectly accurate, ignoring all predictive uncertainty. It simply plugs the mean prediction of the model into an optimizer like MPC.

• Risk: Highly susceptible to model error and compounding error, often leading to catastrophic failures when the agent encounters states where its model is wrong.
• Antithesis: Modern robust MBRL explicitly avoids this by incorporating uncertainty quantification from Bayesian Neural Networks (BNNs) or ensembles to guide pessimistic exploration or weight the trust in model predictions.
• Horizon Limitation: The folly of certainty-equivalence becomes more dangerous as the planning horizon increases, due to the greater opportunity for error accumulation.