Hierarchical Policy

A hierarchical policy introduces temporal abstraction by operating at different time scales. A high-level policy (or manager) selects sub-policies or skill primitives that execute for extended durations, issuing commands every N timesteps. In contrast, the low-level policy (or worker) executes dense, high-frequency motor commands (e.g., joint torques) at every timestep. This structure allows the system to plan over long horizons without requiring the high-level policy to micromanage every instantaneous action.

• Example: A high-level policy might issue the skill "open drawer" once. The low-level "open drawer" sub-policy then executes the precise sequence of reaching, grasping, and pulling actions over hundreds of control cycles.

The architecture is built around a library of reusable skill primitives (e.g., reach, grasp, place, push). The high-level policy's role is to sequence these pre-trained skills to solve novel tasks. This promotes:

• Sample Efficiency: Skills can be trained in isolation on focused datasets or in simulation.
• Generalization: A skill like grasp can be reused across many different high-level tasks ("make coffee," "tidy room").
• Interpretability: The agent's plan is decomposable into human-understandable sub-goals.

This modularity is central to hierarchical reinforcement learning (HRL) and imitation learning frameworks like Option-Critic or HIRO.

Each level of the hierarchy conditions its decisions on different representations of state and goal information.

• High-Level Policy: Conditions on a abstract state representation (e.g., object positions, task progress) and a language instruction (e.g., "unload the dishwasher"). It outputs symbolic sub-goals or skill identifiers.
• Low-Level (Skill) Policy: Conditions on raw or processed sensor data (e.g., images, proprioception) and the specific sub-goal passed down from above (e.g., target end-effector pose for place).

This separation allows the high level to reason symbolically while the low level handles precise, perception-driven control.

Hierarchical policies can be trained end-to-end, but are often trained using a staged or hybrid approach to overcome the credit assignment problem across time scales.

1. Skill Pre-Training: Low-level skills are trained via demonstrations (imitation learning) or goal-conditioned RL in constrained environments.
2. High-Level Policy Training: The manager is trained using RL or imitation learning to select skills, treating the pre-trained skill execution as a "black box" that transitions the environment.
3. Fine-Tuning: Joint fine-tuning may be performed to adapt skills to the specific sequences required by the high-level task.

Frameworks like DiMS (Director-Motor-System) and HAC (Hierarchical Actor-Critic) formalize these training procedures.

The interface between hierarchy levels defines two primary architectural patterns:

• Skill-Centric (Symbolic): The high-level policy selects from a discrete set of skill names (e.g., skill_id = 3). The selected skill primitive then executes its fixed or parameterized policy.
• Sub-Goal-Centric (Continuous): The high-level policy outputs a continuous sub-goal vector (e.g., a desired object position or end-effector pose). A goal-conditioned low-level policy is then trained to achieve any sub-goal within a range. This offers greater flexibility but is harder to train.

Many modern systems, such as those using VLA (Vision-Language-Action) models, use a hybrid where a language model outputs skill names parameterized with arguments (e.g., pick(red_block)).

Compared to a single, flat policy mapping pixels directly to torques, hierarchical policies provide key engineering benefits:

• Long-Horizon Credit Assignment: Simplifies learning by breaking sparse reward problems into shorter-horizon skill achievements.
• Improved Exploration: The high-level policy explores in the space of skills, which is smaller and more meaningful than the space of raw actions.
• Composability & Zero-Shot Generalization: New tasks can often be solved by recombining existing skills in novel sequences without retraining low-level policies.
• Safety & Verifiability: Skill primitives can be individually verified and constrained (e.g., with action masking), and failing sub-tasks can be identified and retried.

This makes them the dominant architecture for complex embodied AI and robotic manipulation tasks.

A foundational reinforcement learning framework for learning hierarchical policies end-to-end. It formalizes the high-level policy as selecting among temporally extended actions called options. Each option is a sub-policy that runs until a termination condition, after which the high-level policy selects a new option. This architecture enables temporal abstraction without manual specification of sub-task boundaries.

• Key Mechanism: The high-level intra-option policy and low-level option policies are trained jointly using policy gradient methods.
• Example: A robot learning to 'make coffee' might have options for 'navigate to kitchen', 'grasp mug', and 'operate machine', each with its own internal policy.

The MAXQ value function decomposition provides a mathematically grounded method for hierarchical policy learning. It breaks down the overall value function of a task into a hierarchy of smaller value functions for each sub-task. The high-level policy is a task graph that calls child sub-tasks.

• Decomposition: The total Q-value is decomposed into the sum of a sub-task's completion value and the Q-values of its child actions.
• Implementation: Used in domains like robot navigation, where a root task 'deliver item' decomposes into 'plan path', 'avoid obstacles', and 'handover object' sub-tasks, each with its own learned policy.

This approach uses a library of pre-learned skill primitives (e.g., 'reach', 'grasp', 'turn valve') as the low-level sub-policies. A high-level sequencing policy, often a transformer or LSTM, learns to select and chain these skills from demonstration data to accomplish long-horizon tasks.

• Process: The high-level policy outputs a sequence of skill identifiers and parameters (e.g., grasp pose). The low-level skill policy then executes the parameterized primitive.
• Real-World Use: In industrial assembly, a high-level policy sequences 'pick circuit board', 'insert component', and 'solder' skills, each implemented as a robust, pre-trained visuomotor policy.

These policies use natural language instructions to guide both high-level planning and low-level execution. A high-level language interpreter parses an instruction (e.g., 'Put the apple on the plate') into a plan of sub-goals or skills. Low-level goal-conditioned policies then execute each sub-goal.

• Architecture: Often uses a large language model (LLM) as the high-level planner to generate a sequence of sub-task commands from text.
• Example: The SayCan paradigm for robots, where an LLM scores the feasibility of proposed skills ('pick apple', 'move to plate') based on current context, and a low-level policy executes the highest-scoring skill.

A modern approach where a high-level diffusion model generates a sequence of intermediate latent goals or states. A separate, trained low-level policy (which could also be a diffusion model) is then conditioned on each latent goal to produce the precise motor actions to reach it.

• Mechanism: The high-level planner operates at a coarse temporal resolution, generating a sparse trajectory of keyframes. The low-level diffusion policy performs fine-grained, high-frequency control between these keyframes.
• Benefit: Provides smooth and diverse long-horizon behavior generation, effective in dexterous manipulation tasks requiring precise sequential contact.

A classical but highly reliable implementation where the high-level policy is a finite-state machine (FSM) or behavior tree. Each state/node represents a discrete sub-policy or skill (e.g., 'Search', 'Align', 'Insert'). Transitions between states are triggered by perceptual predicates or task completion events.

• Structure: Provides deterministic, interpretable control flow. The low-level policies within each state can be classical controllers (PID, IK) or learned networks.
• Industrial Application: Ubiquitous in automated warehousing for mobile robots, where an FSM orchestrates 'navigate to bin', 'scan QR code', 'extend lift', and 'retract' sequences based on sensor feedback.

A skill primitive is a reusable, short-horizon policy or sequence of actions that accomplishes a specific sub-task. In a hierarchical policy, the high-level controller selects from a library of these primitives (e.g., 'open drawer', 'pick up screw'). They abstract low-level control, enabling efficient planning and transfer across tasks.

• Key Feature: Encapsulates robust, low-level behavior.
• Example: A 'peg insertion' primitive handles the fine alignment and force feedback, while the high-level policy sequences 'pick up peg' then 'execute insertion'.

A motion primitive is a parameterized, fundamental movement pattern that serves as a building block for robot trajectories. Unlike skill primitives, they are often purely kinematic or dynamic templates (e.g., a minimum-jerk reaching arc, a wiping oscillation). Hierarchical policies can parameterize and chain these primitives to generate smooth, physically plausible motions.

• Key Feature: Defines the shape of a movement.
• Implementation: Often represented as dynamical systems or via Dynamic Movement Primitives (DMPs).

A goal-conditioned policy is a function that outputs actions based on the current state and a specified goal representation. This is the fundamental unit of a hierarchical level: a high-level policy outputs sub-goals (e.g., 'object position'), and a low-level, goal-conditioned policy executes actions to achieve it. It enables versatility and generalization to new objectives.

• Key Feature: π(a | s, g) where 'g' is the goal.
• Role in Hierarchy: Low-level policies are typically goal-conditioned on the sub-goals provided by the level above.

Task and Motion Planning is a classical hierarchical approach that combines discrete task planning (symbolic reasoning over actions) with continuous motion planning (geometric path finding). It explicitly reasons about symbolic preconditions/effects and feasible kinematics. Modern hierarchical policies often learn to approximate TAMP's reasoning end-to-end.

• Key Feature: Explicit integration of logic and geometry.
• Contrast with Learned Policies: TAMP is often model-based and search-based, while neural hierarchical policies are model-free and learned.

The options framework is a formalization of temporal abstraction in reinforcement learning. An option is a triple: an initiation set (where it can start), an intra-option policy (the skill), and a termination condition. A high-level policy-over-options selects which option to execute until it terminates. This provides a rigorous mathematical foundation for hierarchical reinforcement learning.

• Key Components: Initiation set, policy, termination condition.
• Relation to Hierarchical Policy: A hierarchical policy can be viewed as implementing a policy-over-options, where options are skill primitives.

A manager-worker architecture is a specific neural implementation of hierarchy. A manager network operates at a coarse temporal scale, setting high-level goals. A worker network operates at a fine temporal scale, producing actions to achieve the current goal. The goals act as an attentional mechanism, directing the worker's focus. This is commonly used in environments requiring long-term credit assignment.

• Temporal Decoupling: Manager updates every N timesteps; worker updates every timestep.
• Communication: Goals are often communicated via a latent embedding or a sub-task specification.