Hierarchical Imitation Learning

The high-level planner operates at a coarse temporal abstraction. It does not output low-level motor commands. Instead, it selects which temporally extended skill or option the low-level policy should execute next, based on the current high-level state. This decision is typically made every N timesteps, where N defines the skill's duration. The planner is trained to replicate the expert's sequence of skill selections, learning the task's logical structure.

Low-level skill policies are controllers responsible for executing the specific, short-horizon behaviors selected by the high-level planner. Each skill is a policy (e.g., a neural network) that maps proprioceptive and perceptual inputs to low-level actions (e.g., joint torques). Skills can be:

• Pre-defined: Engineered controllers (e.g., grasp, place, move-to-pose).
• Learned: Trained from demonstration segments corresponding to that skill.
• Parameterized: Accept goals from the high-level planner (e.g., "move to coordinates X,Y").

A core challenge is temporal segmentation: automatically discovering the boundaries between skills in a continuous demonstration. Methods include:

• Change-point detection: Identifying shifts in observation statistics.
• Learning latent variables: Using models like Hidden Markov Models (HMMs) or variational autoencoders to infer discrete skill modes.
• Manual annotation: Human labeling of skill types and transitions in the data. The quality of segmentation directly impacts the high-level planner's ability to learn correct transitions.

The set of all available low-level skills forms a skill library. In the Options Framework from hierarchical reinforcement learning, an option is a triple (I, π, β), where:

• I: The initiation set of states where the option can be started.
• π: The option's internal policy (the low-level skill).
• β: The termination condition, a probability of the option ending in a given state. The high-level planner learns a policy over options, activating π until β terminates it, then selecting a new option.

Hierarchical policies require two levels of state representation:

• Low-Level State: Dense, high-frequency sensor data (joint angles, camera pixels, lidar) used by skill policies.
• High-Level State: Abstracted, lower-dimensional features used by the planner. This may include:
  • Skill completion flags (e.g., "object grasped").
  • Symbolic predicates (e.g., "block-on-table").
  • Goal coordinates for the current skill. The interface between levels is critical: the high-level state must contain sufficient information for planning, and the low-level policy must understand the high-level command.

Training can be end-to-end or modular. In modular training, low-level skills are trained first on segmented demonstrations, then frozen. The high-level planner is trained using the now-discretized expert trajectories. In end-to-end training, both levels are trained jointly, often using gradients flowing through the skill policies to the planner. A key advantage is error containment: a mistake in high-level planning (e.g., picking the wrong skill) does not corrupt the execution of other, correctly selected skills, as each low-level policy remains competent within its domain.

HIL excels at tasks requiring a sequence of distinct skills. A high-level policy sequences temporally extended options (e.g., 'pick screwdriver', 'align with screw', 'turn'), while low-level policies execute the precise motor commands.

Examples:

• Industrial Assembly: Building a complex product from components.
• Kitchen Tasks: Preparing a multi-step meal, like making a sandwich (open bread, spread condiment, place ingredients, close).
• Electronics Repair: Disassembling a device, replacing a part, and reassembling.

Driving is inherently hierarchical. HIL can learn a policy where the high level selects macro-maneuvers (e.g., 'change lane', 'merge onto highway', 'negotiate intersection'), and the low level handles continuous control (steering, throttle, braking).

This structure:

• Improves generalization by reusing low-level skills across many high-level contexts.
• Enables interpretability, as the current high-level option explains the agent's intent.
• Facilitates safe testing by constraining low-level policies to known, demonstrated behaviors.

Complex medical procedures are perfect for hierarchical decomposition. Surgeons demonstrate tasks like laparoscopic suturing, which HIL can break into surgical primitives (knot tying, needle driving, tissue handling).

Key Benefits:

• Skill Modularity: Low-level policies for stable tool manipulation can be reused across different procedures.
• Assistive Autonomy: The system can execute well-defined sub-tasks (e.g., maintaining tension on a suture) under high-level surgeon supervision.
• Training & Simulation: Provides a structured framework for simulating and practicing procedural steps.

In dynamic fulfillment centers, robots must perform multi-step pick-and-place operations. A hierarchical policy can plan: 'Navigate to Bin A12' → 'Grasp Item X' → 'Place in Tote Y'.

Technical Advantages:

• Robustness to Variation: The high-level policy can re-plan the skill sequence if an item is missing, without retraining low-level grasping.
• Efficient Learning: Low-level visuomotor skills (grasping diverse objects) are learned once from demonstrations and shared.
• Scalability: New tasks (e.g., kitting) are defined by new high-level sequences of existing skills.

HIL enables robots to learn collaborative tasks by imitating human partners. The hierarchy allows the robot to infer the human's high-level intent and select complementary skills.

Example Scenario: Assembling furniture together.

• Human Action: Holds a table leg in position.
• Robot Inference: High-level policy selects 'insert fastener' option.
• Robot Execution: Low-level policy aligns the screwdriver and drives the screw.

This requires learning from third-person observations and managing the embodiment gap between human and robot.

HIL is a powerful tool for sim-to-real transfer. Skills (low-level policies) can be trained to mastery in simulation using domain randomization, while the high-level sequencer is learned from a small set of real-world demonstrations or via reinforcement learning in sim.

Process:

1. Train robust low-level skill policies in varied simulated conditions.
2. Collect a handful of real-world demonstrations of the full task.
3. Use HIL to learn the high-level policy that sequences the simulated skills to match the real-world demo.

This drastically reduces the amount of costly real-world data needed.

Behavior Cloning is the foundational supervised learning approach in imitation learning. A policy is trained via regression to directly map observed states to the expert's demonstrated actions. It is simple and sample-efficient but suffers from compounding error when the agent encounters states not in the training distribution, a key limitation that hierarchical methods often aim to mitigate by decomposing long tasks.

A Mixture of Experts architecture is a modular policy design highly relevant to hierarchical learning. It consists of multiple specialized sub-policies (the 'experts') and a gating network that selects or weights their outputs. In hierarchical imitation, the high-level policy can act as the gating network, selecting which low-level skill (expert) to execute, enabling the learning of complex, multi-modal behaviors from demonstrations.

A Dynamic Movement Primitive is a parametric representation for motor skills, modeling them as stable nonlinear dynamical systems. DMPs are a common choice for the low-level policies in a hierarchical framework because they:

• Are easy to learn from a single demonstration.
• Guarantee stability.
• Can be smoothly adapted to new goals (e.g., a different target position) specified by the high-level policy.

Inverse Reinforcement Learning addresses imitation by inferring the latent reward function that the expert is optimizing, rather than directly copying actions. Hierarchical IRL extends this to infer hierarchical reward structures or options. While IRL is more robust to covariate shift than pure BC, it is typically more computationally complex. Hierarchical Imitation Learning can be seen as a more direct, often more efficient, method for learning the policy structure itself.

Trajectory Optimization is a planning method that computes an optimal sequence of states and actions subject to dynamics and constraints. It is closely related to hierarchical imitation in two ways:

1. Generating Demonstrations: Optimal trajectories can be used as expert data for imitation.
2. Refining Policies: The output of a learned hierarchical policy can be refined using trajectory optimization as a post-processing step to ensure dynamic feasibility and optimality.

One-Shot Imitation Learning is a meta-learning challenge where an agent must perform a new task after seeing just a single demonstration. Hierarchical structures are particularly effective for this. A pre-trained library of low-level skills (primitives) allows the high-level policy to quickly compose a new plan from known components based on the novel demonstration, rather than learning motor control from scratch.