Sample Efficiency

The core metric of sample efficiency is the number of real environment interactions required for an agent to achieve a target level of performance. A highly sample-efficient MBRL agent might reach expert-level performance after thousands of interactions, whereas a comparable model-free agent might require millions. This ratio is directly measured by plotting the agent's cumulative reward against the number of environment steps taken.

Efficiency is achieved by offloading exploration to an internal model. Instead of acting randomly in the real world, the agent uses its learned transition model and reward model to simulate thousands of potential futures (imagined rollouts) for the computational cost of a single model query. Key planning techniques include:

• Model Predictive Control (MPC): Re-planning at each step over a short horizon.
• Trajectory Optimization: Using methods like iLQR to find optimal action sequences.
• Latent Imagination: As in the Dreamer algorithm, training policies entirely within a compressed latent space model.

A sample-efficient agent extracts maximal information from each data point. The learned world model acts as a compact, generalizing representation of environment dynamics. This allows the agent to:

• Interpolate between observed states to understand unseen scenarios.
• Reason about consequences of actions without having tried them.
• Reuse a single logged transition (state, action, next state, reward) to improve the model's accuracy across a region of the state-action space, unlike model-free methods which often use experience only once.

Efficient agents do not explore randomly. They perform model-based exploration, deliberately seeking out states where their internal model's predictions are uncertain. This is enabled by uncertainty quantification techniques:

• Probabilistic Ensembles: Training multiple models; disagreement indicates epistemic uncertainty.
• Bayesian Neural Networks (BNNs): Providing a distribution over model parameters. By targeting high-uncertainty regions, the agent collects data that most efficiently reduces model error, accelerating learning.

A defining challenge for sample efficiency is compounding error, where small inaccuracies in the dynamics model explode over long planning horizons. Efficient MBRL systems manage this through:

• Short-horizon planning (e.g., in MPC) with frequent re-planning from real states.
• Regularization of policies to prevent model-policy co-adaptation, where a policy overfits to its own model's biases.
• Pessimistic exploration in offline RL, constraining the policy to areas where the model is confident.

Sample efficiency is evaluated through standardized benchmarks. Common metrics include:

• Final Performance at N Steps: The average return after a fixed budget of environment interactions.
• Area Under the Learning Curve: The integral of the performance vs. step curve, measuring speed and final performance together.
• Asymptotic Performance: The final performance level, ensuring efficiency doesn't come at the cost of capability. Algorithms like MBPO, Dreamer, and MuZero are typically benchmarked against model-free baselines (e.g., SAC, PPO) on suites like DeepMind Control or OpenAI Gym to quantify their efficiency gains.

A world model is an agent's internal, learned representation that predicts future environmental states and rewards. It acts as a compressed simulator, enabling the agent to plan and conduct imagined rollouts without costly real-world interaction. This is the foundational component for achieving high sample efficiency, as policies can be refined extensively in this internal 'dream' space.

Model error is the discrepancy between a learned dynamics model's predictions and the true environment. This error is the primary challenge in MBRL. Compounding error occurs when these inaccuracies accumulate over the course of a multi-step imagined rollout, leading the agent's internal simulation to diverge into unrealistic states. Managing these errors through uncertainty quantification and robust planning is critical for real-world performance.

This refers to techniques for estimating the predictive uncertainty of a learned model. It distinguishes between:

• Epistemic uncertainty: Uncertainty due to lack of knowledge (reducible with more data).
• Aleatoric uncertainty: Inherent stochasticity in the environment. Methods like Bayesian Neural Networks (BNNs) and probabilistic ensembles provide this estimate, which is essential for robust planning (e.g., avoiding uncertain states) and guiding model-based exploration.

The planning horizon is the number of future time steps an agent considers when simulating trajectories with its internal model. It represents a key trade-off:

• Short horizons are computationally cheap but may miss long-term consequences.
• Long horizons enable better long-term strategy but increase compute cost and exposure to compounding model error. Algorithms like Model Predictive Control (MPC) use a receding horizon, planning a sequence but executing only the first action before replanning.

MBPO is a seminal algorithm that blends model-based and model-free RL. It uses short, imagined rollouts from a learned dynamics model to generate synthetic experience. This large dataset of simulated transitions is then used to train a policy using powerful, sample-inefficient model-free algorithms like Soft Actor-Critic (SAC). This hybrid approach leverages the data-efficiency of a model with the asymptotic performance of model-free methods.

This paradigm pushes sample efficiency to its extreme: learning a policy solely from a static, pre-collected dataset without any online interaction. The agent learns a dynamics model from this offline data and then uses it for planning or to generate a vast amount of synthetic experience for policy training. A major challenge is extrapolation error, often addressed via pessimistic exploration techniques that penalize actions in states where the model is uncertain.