Hierarchical Reinforcement Learning

The Options Framework formalizes temporally extended actions as a triple: an initiation set, an intra-option policy, and a termination condition. An option executes until termination, creating a semi-Markov Decision Process (SMDP). This provides a mathematical foundation for HRL, separating high-level skill selection from low-level control.

• Key Mechanism: The agent learns a policy-over-options that selects which skill to invoke, and each option learns its own internal policy.
• Example: A navigation robot might have options for "navigate to kitchen," "pick up object," and "return to base," each lasting hundreds of primitive steps.

Feudal Reinforcement Learning (FRL) organizes learning into a strict management hierarchy. A high-level manager sets subgoals for a lower-level worker over extended time intervals. The worker is rewarded for achieving these subgoals, not the final task reward, creating a clear credit assignment structure.

• Key Mechanism: The manager operates at a slower time scale, issuing goals in a learned or fixed representation space. The worker learns a goal-conditioned policy.
• Example: In a strategy game, a manager might set the subgoal "secure region X," and the worker executes the primitive actions (move, attack) to achieve it.

Hierarchies of Abstract Machines (HAMs) are a programming-inspired approach where the hierarchy is defined by a set of finite-state machines. Each machine calls sub-machines or executes primitive actions. The hierarchy is often designed by a human, providing strong inductive bias and reducing the search space.

• Key Mechanism: Learning focuses on choosing the right path through the pre-defined machine network, rather than inventing skills from scratch.
• Example: A HAM for a delivery robot might have states for "Idle," "Acquire Package," "Plan Route," and "Execute Delivery," each calling lower-level routines.

The MAXQ framework decomposes the overall value function of a task into a sum of value functions for individual subtasks. It uses a recursive optimality criterion, where a policy is optimal for a parent task if it is optimal for each of its child tasks.

• Key Mechanism: Enables task-specific termination and state abstraction, where a subtask may ignore irrelevant parts of the global state. This dramatically improves sample efficiency.
• Example: In a restaurant task, the "serve customer" task's value is decomposed into the values of "take order," "cook food," and "deliver food," each with its own abstracted state representation.

This paradigm focuses on unsupervised skill discovery, where an agent uses intrinsic motivation to autonomously learn a repertoire of useful skills (options) before tackling a specific downstream task. Skills are learned to be diverse and maximize empowerment or coverage of the state space.

• Key Mechanisms: Diversity-based objectives (e.g., maximizing mutual information between skills and states) or curiosity-driven exploration are used to form distinct skills.
• Example: A robot in an empty room might self-discover skills like "rotate," "move forward," and "move backward" without any external reward, which can later be sequenced for a goal like "reach the door."

This modern architecture combines goal-conditioned RL with hierarchy. A high-level policy sets intermediate goals in a learned latent space or actual state space at a low frequency. A low-level, goal-conditioned policy is trained to achieve any goal issued by the high-level policy within a few steps.

• Key Mechanism: Decouples long-term planning (high-level) from short-term, robust control (low-level). Often trained with hindsight experience replay (HER) for the low-level policy.
• Example: For block stacking, the high-level policy might set goals for the position of each block sequentially, while the low-level policy learns the motor commands to move the gripper to any specified position.

HRL is foundational for robotic manipulation and mobile navigation, where tasks require long sequences of primitive actions. A high-level policy might plan a sequence of skills (e.g., 'approach object', 'grasp', 'place'), while low-level controllers execute the motor commands. This abstraction is critical for sim-to-real transfer, as high-level plans can be robust to low-level execution noise. Applications include:

• Warehouse automation: Orchestrating pick-and-place sequences.
• Autonomous vehicles: Hierarchically managing route planning, lane changing, and obstacle avoidance.
• Humanoid robots: Coordinating locomotion and manipulation over extended time horizons.

In complex strategy environments like StarCraft II or Dota 2, HRL agents operate at multiple levels of temporal abstraction. A meta-controller might set strategic goals ('secure resources', 'launch an attack'), while subordinate managers and options handle the tactical micro-management of units over hundreds of game steps. This mirrors military command hierarchy and enables efficient exploration and credit assignment over extended episodes that would be intractable for flat RL agents.

HRL frameworks structure multi-turn dialogue into hierarchical policies. A top-level policy determines the dialog's goal or phase (e.g., 'greet user', 'collect information', 'resolve issue'), while lower-level policies generate the specific utterances or API calls to fulfill that phase. This provides a natural structure for task-oriented dialog systems, allowing for coherent, long-conversation management and easier integration with external tools and knowledge bases.

Managing a global supply chain involves nested decision-making over different timescales and organizational levels. An HRL agent can have a high-level policy that makes quarterly inventory allocation decisions, mid-level policies that handle weekly transport routing, and low-level policies controlling real-time warehouse robot fleets. This decomposition allows the system to react to disruptions (like port delays) by adjusting high-level plans, which then propagate down the hierarchy.

Personalized treatment plans can be modeled as a hierarchical process. The highest level might decide on a therapeutic strategy (e.g., chemotherapy regimen). The next level could manage dosage scheduling over weeks. The lowest level might handle symptom management and adjuvant therapy adjustments in response to daily patient feedback. This structure allows the AI to plan long-term outcomes while adapting to short-term patient responses, a core challenge in clinical workflow automation.

Financial trading systems benefit from temporal abstraction. A top-level portfolio manager could set asset allocation targets for the quarter. A mid-level sector rotation policy adjusts weekly. A low-level execution policy handles minute-by-minute order routing and market making. This separates slow macroeconomic strategy from fast market microstructure tactics, improving data efficiency and stability. HRL helps manage the non-stationarity of markets by allowing strategies at different levels to adapt at their own pace.

The Options Framework formalizes temporal abstraction in HRL. An option is a temporally extended action, defined by a triple: an initiation set (states where it can start), an intra-option policy (how to act), and a termination condition (when it stops). This allows an agent to operate over extended time horizons using macro-actions, significantly reducing the effective planning horizon. For example, a robot's 'navigate to kitchen' option encapsulates thousands of low-level motor commands.

MaxQ is a foundational HRL algorithm that decomposes the overall value function of a task into the sum of value functions for its subtasks. It uses a task graph where parent nodes represent composite tasks and leaf nodes represent primitive actions. The value of a parent task is recursively defined by the values of its children. This decomposition enables:

• State abstraction: Subtasks may ignore irrelevant parts of the state.
• Subtask reuse: Learned subtask policies can be shared across multiple higher-level tasks.
• Theoretical guarantees: It provides a form of hierarchical credit assignment.

Feudal Reinforcement Learning is an HRL architecture inspired by managerial hierarchies. A manager agent operates at a high level, setting subgoals in a abstracted state space over extended time periods. A worker agent learns to achieve these subgoals using primitive actions. The manager is rewarded based on the worker's success. This separation of concerns enforces a commitment horizon, where the manager's directives persist, forcing the emergence of useful skills and providing a natural form of exploration through subgoal specification.

Hierarchical Abstract Machines (HAMs) are a programming language-inspired approach to HRL. A HAM is a finite-state machine where states correspond to calling other machines or choosing primitive actions. This provides a partial program that constrains the space of possible policies, injecting prior knowledge about task structure. The learning agent then fills in the unspecified choices (e.g., which primitive action to take in a given machine state). This framework guarantees that learned policies adhere to the specified hierarchical constraints, improving safety and sample efficiency.

Skill Discovery refers to the automatic learning of reusable temporally-extended actions (skills or options) without explicit human specification. Methods often use intrinsic motivation signals, such as:

• Empowerment: Learning skills that maximize an agent's control over its future states.
• Diversity: Encouraging a set of skills to be distinct and cover different regions of the state space.
• State visitation novelty: Learning skills that reach new states. Once discovered, these skills form a skill library that can be used by a higher-level policy to solve complex tasks more efficiently, a key step towards autonomous hierarchical learning.

A Semi-Markov Decision Process (SMDP) is the mathematical framework underlying the Options Framework. It generalizes a Markov Decision Process (MDP) by allowing actions to take a variable amount of time to complete. In the HRL context, each option is an action in the SMDP. The SMDP's transition probability describes the probability of landing in a new state after an option terminates, and its reward function accumulates the discounted sum of primitive rewards over the option's execution. Planning and learning algorithms (like SMDP Q-learning) can then operate over options, treating them as atomic units.