Hierarchical Reinforcement Learning (HRL)

The Options Framework is the foundational formalization for HRL, introducing the concept of temporally extended actions. An option is defined as a triple (I, π, β) where:

• I is the initiation set: states where the option can be started.
• π is the intra-option policy: a lower-level policy that selects primitive actions.
• β is the termination condition: a probability of the option terminating in each state. This framework allows the high-level policy to select among options, which then execute until termination, abstracting away sequences of primitive steps. It provides a mathematically sound way to incorporate temporal abstraction into the standard MDP formulation.

The Hierarchy of Abstract Machines is a programming-inspired HRL model that constrains an agent's policy through a hierarchy of finite-state machines. Each machine:

• Operates at a different level of temporal abstraction.
• Calls lower-level machines or executes primitive actions.
• Has internal states that guide its execution flow. Unlike the Options Framework, HAMs impose structural constraints on the policy space, which can drastically reduce the search complexity for the learning agent. This makes them particularly useful for imposing prior knowledge about task decomposition and safe behavior patterns in robotics.

MaxQ is a method for decomposing the overall value function of a task into the sum of value functions for individual subtasks. It breaks the Bellman equation hierarchically. Key elements include:

• Root Task: The overall objective (e.g., "deliver coffee").
• Subtasks: Decomposed goals (e.g., "navigate to kitchen," "grasp mug").
• Terminal Predicates: Conditions that define when a subtask is complete. The projected value function for a subtask is the value of executing its policy until termination, ignoring rewards from lower-level subtasks. This decomposition enables off-policy learning within subtasks and facilitates skill transfer, as subtask policies can be learned independently and reused in different root contexts.

Feudal Reinforcement Learning structures the learning hierarchy like a medieval feudal system, where a manager (high-level) sets goals for a worker (low-level). The core mechanism is the goal or command space.

• The manager learns a policy to set goals in a learned or predefined abstract space (e.g., a spatial coordinate or a latent vector).
• The worker learns a policy to achieve the current goal using primitive actions.
• The manager rewards the worker based on whether the goal was achieved, not on the final external reward. This separation of concerns allows the manager to plan over long horizons using abstract goals, while the worker becomes a specialized, goal-conditioned skill. It is a precursor to modern goal-conditioned HRL methods.

This component focuses on automatically learning the hierarchy rather than designing it manually. Methods include:

• Intrinsic Motivation: Using curiosity or empowerment to drive the discovery of useful, reusable skills. An agent might be rewarded for reaching novel states or learning skills that maximize control over future states.
• Subgoal Discovery: Algorithms that identify bottleneck states (states that connect different regions of the state space) or use clustering in the state space to propose candidate subgoals for higher-level policies.
• Skill Chaining: Automatically sequencing discovered skills to solve longer tasks. This transforms HRL from a manually engineered framework to a fully learned, emergent property of the agent's interaction with the environment.

Modern deep HRL implementations often use goal-conditioned hierarchical policies. The architecture typically consists of two neural networks:

• High-Level (Manager) Policy: Operates at a lower frequency (every k timesteps). It observes the state and outputs a goal (e.g., a vector in a latent space or a region of state space).
• Low-Level (Worker) Policy: A goal-conditioned policy that receives the current state and the current goal from the manager. It outputs primitive actions at every timestep and is trained to achieve the given goal.
• Goal Representation: A critical design choice, often a learned embedding. The worker is trained with an intrinsic reward based on reaching the goal (e.g., negative distance to the goal state). This architecture is the practical implementation of feudal RL and is central to algorithms like HIRO (Data-Efficient Hierarchical RL) and HAC (Hierarchical Actor-Critic).

A classic HRL application is training a robotic arm to prepare a meal. The high-level policy might select macro-actions like [MakeSandwich, BrewCoffee]. The MakeSandwich subtask policy is itself a hierarchy:

• Skill 1: Navigate to fridge (low-level locomotion).
• Skill 2: Open door and grasp ingredients (manipulation).
• Skill 3: Assemble components on counter (fine motor control). Each skill is a reusable option or temporally extended action, allowing the robot to learn the overall task without reasoning over every individual joint movement from the start.

In logistics, an Autonomous Mobile Robot (AMR) uses HRL for efficient navigation and task completion in a dynamic warehouse.

• Top Level (Manager): Receives order: "Retrieve item A from zone 12."
• Mid Level (Controller): Executes the plan: [NavigateToZone12, LocateShelf, PickItem, ReturnToDock].
• Low Level (Primitive Actions): Each controller activates atomic motor commands (e.g., move_forward(0.5m), turn(90deg)). This hierarchy allows the AMR to re-plan at the mid-level if a path is blocked, without discarding the entire high-level mission, demonstrating robust temporal abstraction.

The Options Framework is a foundational HRL formalism that defines a temporally extended action as a triple (I, π, β).

• Initiation Set (I): The set of states where the option can be started.
• Intra-Option Policy (π): The policy that selects primitive actions while the option is executing.
• Termination Condition (β): The probability the option terminates in any given state. In this model, the agent learns a policy over options at a higher level and policies within options at a lower level. This creates a semi-Markov Decision Process (SMDP), where decisions are made at the option level, drastically reducing the planning horizon.

Feudal RL is an HRL approach inspired by managerial hierarchies, where a manager module sets goals for a worker module.

• The manager operates at a slower time scale, setting abstract, high-dimensional goals (e.g., "go to the north-east quadrant").
• The worker operates at a faster time scale, translating these goals into low-level actions to achieve them.
• The manager learns by observing the worker's success and is rewarded for setting achievable goals. This explicit separation of goal-setting and goal-achievement facilitates learning in vast state spaces and enables transfer, as a worker trained on basic navigation can be directed by new managers for different high-level tasks.

A major advantage of HRL is unsupervised skill discovery and subsequent transfer learning.

• Algorithms like DIAYN (Diversity is All You Need) or HIRO (Data-Efficient Hierarchical RL) can learn a library of useful skills (e.g., open door, push object, move quickly) without a task-specific reward.
• These skills become reusable primitives for new, complex tasks. For example, a robot that has mastered push object in simulation can transfer that skill to a real-world task like clearing a path, significantly accelerating sim-to-real transfer and reducing the sample complexity for learning the new composite task.

MAXQ is a hierarchical value function decomposition method that breaks the overall value of an action into the value of its subtasks.

• It represents the task hierarchy as a directed acyclic graph of subtasks.
• The value of a parent task is decomposed into the completion value of its child tasks.
• For example, the Q-value for the high-level action NavigateToRoom is decomposed into the sum of values for ExitCurrentRoom, TraverseHallway, and EnterTargetRoom, plus their intrinsic rewards. This decomposition allows for state abstraction within subtasks (a subtask may ignore irrelevant state variables) and enables more efficient learning and theoretical analysis of the hierarchy.

The Options Framework is a formalization of temporal abstraction in HRL. An option is a temporally extended action, defined by a triple:

• Initiation Set: States where the option can be started.
• Policy: The low-level policy that selects primitive actions.
• Termination Condition: A probability of terminating in each state.

This framework allows an agent to operate at multiple time scales, where a high-level policy selects among options, and each option executes a sequence of primitive actions until termination. It is the mathematical backbone for many HRL algorithms, enabling the learning of reusable skills.

Feudal Reinforcement Learning is an early HRL architecture inspired by managerial hierarchies. It introduces a manager module and a worker module.

• The manager operates at a higher level of abstraction and a slower time scale. It sets goals or subgoals for the worker in a goal space that is simpler than the raw state space.
• The worker learns a policy to achieve the goals set by the manager using primitive actions.

This separation of concerns simplifies learning by allowing the manager to focus on long-term strategy while the worker handles short-term control, directly addressing the credit assignment problem over long time horizons.

MAXQ Value Function Decomposition is a method for decomposing the overall value function of a task into a hierarchy of smaller value functions for each subtask. It breaks down the projected value function of a parent task into the sum of:

• The completion function (value of completing the child subtask).
• The action-value function of the child subtask itself.

This recursive decomposition enables off-policy learning within the hierarchy and provides a principled way to share learned subtasks across multiple parent tasks. It guarantees that an optimal policy for the hierarchical decomposition is also optimal for the original flat MDP.

Skill Discovery refers to the autonomous learning of a repertoire of useful skills or options without explicit human specification of the subtask hierarchy. Common approaches include:

• Diversity-Driven Exploration: Maximizing the diversity of visited states or outcomes to encourage the emergence of distinct behaviors.
• Mutual Information Maximization: Learning skills that maximize the mutual information between the skill identifier and the resulting state distribution.
• Unsupervised Pre-training: Learning skills in a reward-agnostic manner, creating a library of primitives that can later be composed by a high-level policy to solve specific downstream tasks.

This is critical for scaling HRL to complex domains where manual engineering of the hierarchy is infeasible.

Hierarchical Abstract Machines (HAMs) are a programming language-inspired approach to HRL. A HAM is a finite-state machine that constrains the agent's policy, reducing the effective size of the policy search space.

• States in the machine correspond to calling sub-machines or choosing primitive actions.
• The learning problem is reduced to finding the optimal policy within the constrained space defined by the HAM.

Unlike the Options Framework, HAMs can enforce hard constraints on behavior (e.g., always recharge before executing a power-intensive skill). They provide a bridge between programmed logic and learned control, useful for embedding safety or procedural knowledge.

Goal-Conditioned Hierarchical RL structures the hierarchy around achieving specified goals. A high-level policy selects a subgoal (e.g., a desired state or feature vector) for a lower-level goal-conditioned policy to achieve.

• The lower-level policy is trained to reach a wide variety of goals, making it a general-purpose skill.
• The high-level policy learns to sequence these subgoals to accomplish the overall task.

This paradigm is highly effective for long-horizon robotic manipulation tasks (e.g., "make coffee") where the final goal can be decomposed into subgoals like "grasp cup," "navigate to machine," and "press button." It naturally facilitates transfer learning, as the goal-conditioned low-level policy can be reused for new high-level tasks.