Hierarchical Reinforcement Learning (HRL)

Temporal abstraction is the fundamental concept in HRL that allows an agent to operate over extended time horizons without making decisions at every primitive time step. It introduces higher-level actions, called options or skills, which are policies that execute for a variable number of steps before terminating. This enables the agent to plan and learn at multiple levels of granularity, dramatically improving sample efficiency and planning speed in long-horizon tasks like robotic manipulation or navigation.

The Options Framework formalizes temporal abstraction. An option is a triple (I, π, β) consisting of:

• Initiation Set (I): The set of states where the option can be started.
• Policy (π): The internal policy that selects primitive actions until termination.
• Termination Condition (β): The probability of terminating in each state. At the high level, the agent selects among options (e.g., 'navigate to kitchen'), and the selected option's policy executes low-level actions (e.g., motor commands) until it terminates, at which point the high-level policy selects a new option. This creates a semi-Markov Decision Process (SMDP).

Skill discovery is the process of automatically learning a useful set of temporally extended actions (skills) without explicit human specification. Common approaches include:

• Diversity-Driven Intrinsic Motivation: Encouraging the agent to discover skills that lead to diverse states, maximizing coverage of the state space.
• Variational Autoencoders (VAEs): Learning a latent space of skills where different latent vectors correspond to different emergent behaviors.
• Graph-Based Methods: Clustering states visited in trajectories to identify meaningful subgoals. This automation is critical for scaling HRL to complex domains where manually designing subtasks is infeasible.

HRL systems implement a hierarchical policy architecture, typically with two or more levels:

• Manager (High-Level): Operates at a coarse time scale, sets subgoals or selects options for the worker. It might plan every N primitive steps.
• Worker (Low-Level): Receives the subgoal from the manager and learns a policy to achieve it using primitive actions. It operates at the native environment time step. This separation of concerns allows the manager to learn long-term strategy while the worker specializes in short-horizon control. Architectures like FeUdal Networks and HIRO are prominent examples.

A key challenge in HRL is how the high-level manager communicates intent to the low-level worker. This is often done via subgoal specification. The manager does not output primitive actions but instead outputs a goal representation (e.g., a target joint angle or image feature). The worker then receives an intrinsic reward based on how close it gets to this subgoal, independent of the sparse external task reward. This creates a dense, curriculum-like learning signal for the worker and allows the manager to learn using the external reward, effectively credit assigning success or failure across the hierarchy.

Modern HRL algorithms leverage off-policy learning to improve data efficiency. Since experiences are generated by a mixture of high- and low-level policies, they can be stored in a shared replay buffer. Both levels can then learn from this buffer, reusing experiences generated by past policies. Algorithms like HIRO and SNN4HRL use off-policy corrections to allow the high-level policy to learn from data collected under an older version of itself, which is essential for stable training in complex, sparse-reward environments common in robotics and control.

In complex strategy games like StarCraft II, HRL agents operate at multiple time scales. A meta-controller sets strategic goals (e.g., 'build economy', 'launch attack'), while subordinate managers and workers execute the tactical and micro-level actions. This mirrors human command hierarchies and provides:

• Abstract strategic reasoning over long time horizons
• Parallel execution of different sub-tasks (economy and military)
• Robustness to low-level tactical failures DeepMind's FTW agent for Quake III and AlphaStar for StarCraft II utilize hierarchical architectures to master these multi-scale decision-making environments.

Self-driving systems use HRL to manage the hierarchy of driving tasks. A top-level policy selects a maneuver (e.g., 'change lanes', 'negotiate intersection'), a mid-level policy plans a trajectory, and a low-level controller issues throttle/steering commands. This enables:

• Safe decomposition of the complex driving policy
• Explicit adherence to traffic rules at the high level
• Smooth handling of nested sub-tasks (e.g., lane change within a route) In heterogeneous fleet orchestration, a high-level coordinator assigns macro-tasks to individual vehicles, which then use their own HRL stacks for local navigation and execution.

Feudal RL is a specific HRL architecture inspired by managerial hierarchies. A manager sets goals at a coarse spatial or temporal scale for a worker, which learns to achieve those goals. The manager is rewarded based on the worker's success. This approach provides:

• Natural credit assignment across time scales
• Learning of goal-conditioned sub-policies
• Interpretable sub-goal generation For example, in a navigation task, a manager might command a worker to 'reach the north corridor', and the worker learns the low-level movements to achieve any such directional command. This decouples high-level planning from low-level control.

MAXQ is a hierarchical value function decomposition method. It breaks the total Q-value of a task into the sum of Q-values for its subtasks and a completion function. This recursive decomposition allows for:

• Provably optimal hierarchical policies under certain conditions
• State abstraction within subtasks, ignoring irrelevant variables
• Efficient learning as subtasks can be learned and evaluated independently For instance, in a delivery robot task, the Q-value for 'deliver package' decomposes into values for 'navigate', 'pickup', and 'drop-off' subtasks, plus the reward for completing the delivery. This structure is used in logistics and autonomous supply chain intelligence systems.

A Markov Decision Process (MDP) is the foundational mathematical framework for modeling sequential decision-making. It is defined by a tuple (S, A, P, R, γ) of states, actions, transition probabilities, rewards, and a discount factor. HRL extends the flat MDP by introducing a hierarchy of policies, where higher-level policies select among temporally extended options or skills, and lower-level policies execute the primitive actions within those options. This decomposition creates a semi-Markov Decision Process (SMDP), where the time between high-level decisions is variable.

The Options Framework is a formalization of temporal abstraction central to many HRL approaches. An option is a triple (I, π, β):

• Initiation Set (I): States where the option can be started.
• Policy (π): The internal policy that selects primitive actions.
• Termination Condition (β): Probability of terminating in each state.

Higher-level meta-policies select among options, which execute until termination. This allows the agent to operate at multiple temporal scales, bypassing the need to plan at the level of every motor command. The framework provides a clean interface for integrating learned skills into planning.

Feudal Reinforcement Learning is a seminal HRL architecture inspired by managerial hierarchies. It features a manager module and a worker module. The manager operates at a coarse time scale, setting subgoals in a learned or engineered abstract state space. The worker's objective is to achieve these subgoals by taking primitive actions. The manager provides an intrinsic reward to the worker based on subgoal achievement. This separation of concerns enforces a goal-conditioned hierarchy, where high-level planning is decoupled from low-level control, improving exploration and transfer.

Skill Discovery refers to the unsupervised or intrinsic motivation-driven process of learning a repertoire of reusable skills or options without an extrinsic task reward. Common methods include:

• Diversity-Driven: Maximizing the mutual information between skills and states visited.
• Prediction-Based: Learning skills that lead to predictable changes in the environment.
• Graph-Theoretic: Identifying bottlenecks in state-transition graphs.

These discovered skills form the primitive library for hierarchical planning. When a new task is presented, the HRL agent can compose these pre-trained skills, dramatically accelerating learning through transfer and reducing the search space.

MAXQ Value Function Decomposition is a formal method for decomposing the value function of a hierarchical policy into the sum of values for individual subtasks. It breaks the overall Q-function Q(s, a) into:

• Completion Function: The value of the parent task after the child subtask finishes.
• Subtask Value Function: The value of executing the child subtask itself.

This recursive decomposition allows for off-policy learning within the hierarchy and enables subtask policies to be learned independently and reused across different parent tasks. It provides a mathematically grounded way to implement hierarchical credit assignment.

A Hierarchical Actor-Critic architecture extends the standard actor-critic framework to multiple levels. Typically, it consists of:

• Meta-Actor (Manager): A high-level policy that outputs goals or selects skills.
• Sub-Actor (Worker): A low-level, often goal-conditioned policy that executes actions.
• Hierarchical Critics: Value functions at each level that critique the choices of their respective actors.

Training can be end-to-end using gradients flowing through the hierarchy, or levels can be trained separately. This architecture is the backbone of modern deep HRL algorithms like HIRO (Data-Efficient Hierarchical Reinforcement Learning) and HAC (Hierarchical Actor-Critic).