World Model

The state representation is the compressed, informative encoding of the current environment configuration. It transforms high-dimensional raw sensory inputs (e.g., pixels) into a low-dimensional latent vector that captures the essential information for decision-making.

• Purpose: Serves as the agent's internal 'belief' about the world.
• Key Methods: Include deterministic recurrent networks, stochastic latent variables (as in Recurrent State-Space Models), or object-centric representations that decompose scenes into entities.
• Goal: To be Markovian, meaning the current state is sufficient to predict the future, and often disentangled, separating independent factors of variation.

The dynamics model (or transition model) predicts the next state representation given the current state and a proposed action. It encodes the learned 'physics' or rules of the environment.

• Function: Implements s_{t+1} = f(s_t, a_t), often with a probabilistic output to account for uncertainty.
• Architecture: Typically a neural network trained via prediction error on observed transitions.
• Critical Role: Enables mental simulation or 'imagination.' The agent can roll out possible futures internally without acting in the real world, which is the foundation for planning algorithms like Model-Predictive Control (MPC).

The reward model predicts the expected immediate reward for a given state-action pair. It allows the agent to evaluate the desirability of predicted trajectories during internal planning.

• Training: Learned by regressing against observed rewards r_t = g(s_t, a_t).
• Integration with Planning: During simulated rollouts, the cumulative sum of predicted rewards provides an estimate of a trajectory's value. In algorithms like MuZero, the reward model is learned jointly with value and policy predictions.
• Distinction: In some frameworks, this is separate from the value function, which estimates long-term return.

These components handle the interface between the raw perceptual world and the abstract state representation.

• Encoder: A neural network (e.g., CNN) that compresses high-dimensional observations o_t (like images) into the latent state s_t. In Partially Observable MDPs, it integrates history.
• Decoder: Reconstructs observations from the latent state. Its training loss (reconstruction error) is a primary driver for learning useful state representations that retain salient information.
• Role: They enforce that the latent state contains enough information about the world, making the representation grounded in sensory reality.

This is the algorithmic component that uses the world model (state, dynamics, reward) to select optimal actions. It performs simulation-based search over possible futures.

• Process: Given a current state, the mechanism uses the dynamics and reward models to simulate multiple action sequences (rollouts), evaluates their outcomes, and selects the best initial action.
• Common Algorithms: Include Monte Carlo Tree Search (MCTS) (as used by AlphaZero and MuZero) or gradient-based planners that backpropagate through the model.
• Output: The action with the highest predicted cumulative reward, or a refined policy distribution.

A robust world model must quantify its own epistemic uncertainty—what it does not know due to limited data. This is crucial for safe exploration and robust planning.

• Methods: Ensemble dynamics (training multiple models) is a common approach. Disagreement among ensemble members signals high uncertainty in a state-action region.
• Applications: Guides exploration (visit uncertain states) and enables pessimistic planning in offline RL (avoid actions where the model is unreliable).
• Relation: Connects to concepts like the sim2real gap, where uncertainty estimation can trigger model adaptation.

A class of reinforcement learning algorithms where an agent explicitly learns a model of the environment's dynamics (a world model) and uses it for planning or policy optimization. Unlike model-free methods that learn from trial-and-error, MBRL agents can simulate outcomes internally, leading to greater sample efficiency. Key approaches include using the model for Dyna-style planning, model-predictive control, or to generate synthetic experience for policy training.

A mathematical framework that extends the Markov Decision Process (MDP) for environments where the agent cannot directly observe the true state. The agent must maintain a belief state—a probability distribution over possible states—based on its observation history. World models are often used to implement or approximate this belief state, enabling planning under uncertainty. This is critical for real-world robotics where sensors provide noisy, incomplete data.

A specific, influential world model architecture popularized by the Dreamer series of agents. It combines:

• A deterministic recurrent neural network to track temporal dependencies.
• A stochastic latent variable model to capture uncertainty and multiple futures. This hybrid design learns compact state representations directly from high-dimensional inputs like pixels and enables efficient latent space planning. It is a foundational example of a learned, neural world model.

An online control method that uses a dynamics model (which can be a world model) to predict future states over a finite horizon. At each step, it solves an optimization problem to find the best sequence of actions, executes the first one, and then replans. This receding horizon control makes it robust to model inaccuracies. In robotics, MPC is widely used for locomotion and manipulation, often leveraging learned or analytical world models.

The performance discrepancy between a system trained in simulation and its behavior when deployed in the real world. This gap arises from inevitable inaccuracies in the simulator's physics, visuals, or sensor models. World models trained purely in simulation suffer from this. Techniques to bridge the gap include:

• Domain randomization (varying simulation parameters).
• System identification (calibrating the sim to real data).
• Online model adaptation (updating the world model with real experience).

The process of encoding the current configuration of an environment into a compact, informative format that is sufficient for decision-making. A world model's internal state is a form of state representation. Goals for effective representation include:

• Sufficiency for predicting future states and rewards.
• Disentanglement of independent factors of variation.
• Abstraction, ignoring irrelevant details.
• Causal structure, capturing only variables that influence outcomes. This is distinct from but essential to building a predictive world model.