Hierarchical World Model

The core mechanism of a hierarchical world model is its structured representation at different levels of abstraction. This typically involves:

• High-Level Abstractions: Represent long-term goals, subgoals, and abstract concepts (e.g., 'navigate to the kitchen').
• Mid-Level Abstractions: Represent sequences of actions or object interactions (e.g., 'open door', 'pick up cup').
• Low-Level Abstractions: Represent primitive motor commands or sensory details (e.g., joint angles, pixel values). This structure allows the agent to plan efficiently by reasoning at the appropriate level, avoiding the computational explosion of planning with raw sensory data.

Hierarchical models incorporate temporal abstraction, where high-level actions (often called options or skills) persist over extended time periods before terminating. This is formalized in frameworks like the Hierarchical Reinforcement Learning (HRL) Options Framework. Key features include:

• Initiation Set: States where the high-level action can be started.
• Termination Condition: States where the action ends.
• Internal Policy: The low-level policy that executes until termination. This allows an agent to execute a macro-action like 'make coffee' without micromanaging every muscle twitch, dramatically improving planning horizon and sample efficiency.

Instead of a monolithic state representation, hierarchical models factor the state space according to abstraction level. For example:

• A robot's state might be factored into (room_location, arm_position, gripper_force).
• A high-level planner reasons over room_location to navigate.
• A low-level controller reasons over arm_position and gripper_force to grasp. This factorization is often learned via disentangled representation learning, where a latent vector's dimensions correspond to independent factors like object identity, position, and lighting. This enables modular reasoning and transfer of skills.

Hierarchical planning operates by generating and achieving subgoals. The high-level model produces a sequence of subgoal states (e.g., 'door open', 'cup in gripper'), and low-level controllers are tasked with reaching each subgoal. This decomposes a complex task into manageable chunks. Techniques include:

• Feudal Reinforcement Learning: A manager module sets subgoals for a worker module.
• HIRO (Data-Efficient Hierarchical RL): Uses off-policy correction to learn high-level and low-level policies simultaneously from experience replay.
• Subgoal Testing: The high-level model can mentally simulate reaching subgoals using its learned dynamics before committing to a plan.

The abstraction hierarchy can be discovered automatically through learning. Common approaches include:

• Skill Discovery: Using unsupervised RL or intrinsic motivation to discover frequently useful action sequences, which become reusable skills. Methods like DIAYN (Diversity is All You Need) incentivize learning distinguishable skills.
• Goal-Conditioned Hierarchical RL: The low-level policy is trained to reach any goal within a subspace, while the high-level policy learns to choose which subspace goal to target next.
• Variational Autoencoders (VAEs) for State Abstraction: A VAE can learn to encode raw observations into a latent space where the hierarchy is enforced via the prior, such as a Vector-Quantized VAE (VQ-VAE) creating discrete high-level codes.

Hierarchical world models are a practical implementation strategy for tackling Partially Observable Markov Decision Processes (POMDPs). The hierarchy acts on a belief state—a distribution over possible true states. Higher levels maintain a coarser, more abstract belief. This is critically enabled by digital twin simulations, where the hierarchical model can be trained and tested in a high-fidelity virtual replica. The model learns to predict outcomes at different abstraction levels within the simulation, enabling safe transfer of hierarchical planning strategies to the real world through sim-to-real transfer learning.

A POMDP is the foundational mathematical framework for sequential decision-making under uncertainty, where an agent cannot directly observe the true environment state. It formalizes the need for a world model.

• Belief State: The agent maintains a probability distribution over possible true states, updated via a Bayesian filter.
• Hierarchical Extension: Hierarchical world models often implement Hierarchical POMDPs (HiPOMDPs), where abstract actions at a high level correspond to solving sub-POMDPs at lower levels.
• Application: This framework is essential for modeling real-world robotics and dialogue systems where sensors provide only partial information.

Model-Based RL is a paradigm where an agent learns an explicit model of the environment's dynamics (transition function) and reward function. This learned model is the agent's world model.

• Planning: The agent uses this model for internal simulation (e.g., via Monte Carlo Tree Search) to plan sequences of actions before acting in the real world.
• Hierarchical Planning: A hierarchical world model enables planning at multiple temporal abstractions. High-level plans set long-term subgoals, while low-level models determine the precise actions to achieve them.
• Sample Efficiency: By learning a model, agents can learn from imagined experience, drastically reducing the number of expensive real-world interactions needed.

The Options Framework is a formalization of temporally extended actions (macro-actions) in reinforcement learning. It is a direct precursor and component of hierarchical world models.

• An option is a triple: an initiation set (where it can start), an internal policy (the sequence of primitive actions), and a termination condition.
• Abstraction: A hierarchical world model can be viewed as learning the dynamics and outcomes of these options. High-level reasoning selects which option to execute, abstracting away low-level details.
• Skill Learning: Options represent reusable skills or behaviors. Discovering a useful set of options is a key challenge in hierarchical RL.

Feudal RL is an early hierarchical approach inspired by managerial hierarchies. It explicitly separates planning layers, where a manager sets goals for a worker.

• Goal Transmission: The manager operates at a coarse spatial/temporal scale and communicates abstract goal images or feature targets to the worker.
• Information Hiding: The worker learns to achieve these goals without needing to understand the manager's overall objective, enforcing a clean abstraction barrier.
• Modern Analogue: This architecture is a clear blueprint for modern hierarchical world models used in robotics, where a high-level planner sets subgoal coordinates for a low-level controller.

A Successor Representation (SR) is a neural representation that predicts the expected future occupancy of states. It provides a form of predictive world knowledge that facilitates fast planning and abstraction.

• Temporal Abstraction: Successor Features extend SRs to feature spaces, enabling the calculation of long-term value for new tasks rapidly (generalized policy evaluation).
• Hierarchical Link: In hierarchical models, SRs can be learned at different levels. A high-level SR might predict which abstract states (e.g., rooms) will be visited, while a low-level SR predicts primitive states within a room.
• Efficient Planning: SRs decouple the dynamics of the environment from rewards, allowing for fast re-planning when goals change.

Object-Centric Representations structure a scene as a collection of discrete entities (objects) with attributes like position, shape, and color. This is a powerful inductive bias for hierarchical world models.

• Compositionality: The world model can reason about object interactions (e.g., 'stack', 'contain') by composing object representations, rather than modeling pixels.
• Abstract States: High levels of a hierarchy can operate on object categories or relationships (e.g., 'key is in drawer'), while low levels handle precise poses and physics.
• Generalization: Models built on object representations generalize better to novel configurations and support symbolic planning methods, bridging neural and symbolic AI.