Hierarchical Reinforcement Learning (HRL)

Temporal abstraction allows an HRL agent to operate over extended time horizons by encapsulating sequences of primitive actions into higher-level skills or options. Instead of planning at every timestep, the agent can choose a high-level skill that executes for a variable duration. This is formalized through the options framework, where an option is defined by an initiation set, a policy, and a termination condition. This abstraction dramatically reduces the complexity of the planning problem and enables the agent to ignore irrelevant low-level details when making strategic decisions.

A core advantage of HRL is the reuse and composition of learned skills across different tasks. Once a low-level policy for a subtask (e.g., 'navigate to door', 'grasp object') is mastered, it can be invoked as a building block by higher-level controllers for novel, more complex goals. This compositional structure mirrors human problem-solving and provides significant sample efficiency benefits. The hierarchy allows the agent to mix and match pre-existing skills rather than learning every new task from scratch, accelerating transfer learning.

Hierarchical credit assignment is the process of attributing success or failure (reward) to the correct level in the hierarchy. A high-level manager must learn which subgoal led to a final reward, while a low-level worker must learn which primitive actions achieved that subgoal. This is a non-trivial challenge, as delayed rewards must be propagated through multiple levels of abstraction. Algorithms address this using intrinsic rewards for subgoal achievement or through feudal learning paradigms where managers provide goals and workers are rewarded for achieving them.

A major research focus in HRL is automatic subgoal discovery—how an agent can learn a useful hierarchy without manual engineering. Methods include:

• State visitation statistics: Identifying bottleneck states that connect regions of the state space.
• Skill diversity: Encouraging the discovery of distinct, useful skills through information-theoretic objectives.
• Goal-conditioned policies: Learning universal policies that can reach any specified subgoal, which are then orchestrated by a higher-level planner. Effective subgoal generation creates a set of intermediate milestones that make the overall task tractable.

Feudal Reinforcement Learning is a specific HRL architecture inspired by managerial hierarchies. It features two distinct levels:

• A manager operates at a coarse time scale and in a abstracted state space. It sets high-level goals (or 'directions') for the worker.
• A worker operates at a fine time scale. It learns to achieve the specific goals set by the manager and receives a reward for doing so. Communication is via goal vectors, and the manager is penalized if the worker fails, creating a clear contractual relationship. This separation of concerns simplifies learning at each level.

The options framework (Sutton, Precup, Singh) provides a formal model for temporal abstraction in HRL. An option is a triple (I, π, β):

• I: The initiation set of states where the option can be started.
• π: The option's policy, mapping states to primitive actions.
• β: The termination condition, a probability of stopping in each state. The agent's decision-making process becomes a semi-Markov Decision Process (SMDP), where choices between options are made at option initiation and termination times. This framework allows standard RL algorithms to be extended to the hierarchical setting.

HRL is foundational for complex robotics. A high-level controller might plan a sequence of macro-actions like 'navigate to kitchen' and 'pick up cup', while low-level controllers execute the primitive actions of motor control. This hierarchy enables:

• Skill Abstraction: Reusable low-level skills (e.g., 'grasp', 'move forward') are learned once.
• Long-Horizon Planning: The agent plans over abstract skills, making decades-long action sequences tractable.
• Transfer Learning: Skills learned in simulation can be transferred to physical robots by adapting only the high-level planner. Real-world systems, like Boston Dynamics' robots, utilize hierarchical control architectures for robust locomotion and task execution in unstructured environments.

In games like StarCraft II or Dota 2, HRL agents operate at multiple strategic levels. A meta-controller sets high-level goals (e.g., 'economy boom', 'map control'), while subordinate managers execute tactical routines (e.g., 'build army', 'harass enemy workers').

• Temporal Abstraction: The agent doesn't micromanage every unit click but issues commands valid for hundreds of game steps.
• Modular Strategy: Different high-level policies can be swapped to adapt to an opponent's playstyle.
• Efficient Exploration: The agent explores in the space of strategies, not low-level actions, converging on optimal play faster. DeepMind's AlphaStar utilized hierarchical reasoning to achieve Grandmaster level in StarCraft II, demonstrating superhuman strategic planning.

Self-driving cars use HRL to break down the driving task. A top-level route planner selects a route (e.g., 'take highway exit 42'). A mid-level behavioral planner decides maneuvers (e.g., 'change lanes', 'merge'). A low-level controller executes throttle and steering commands.

• Safety through Hierarchy: Lower-level controllers have built-in safety constraints (e.g., collision avoidance) that operate independently of high-level goals.
• Real-time Decision Making: Planning at the maneuver level (seconds) is computationally feasible, whereas planning at the actuator level (milliseconds) is not.
• Handling Complexity: The hierarchy naturally separates strategic navigation from tactical driving and reflexive control.

The Options Framework is a formalization of HRL within the Markov Decision Process (MDP) theory. An option is a temporally extended action, defined by a triplet: an initiation set (states where it can start), an internal policy (the sequence of primitive actions), and a termination condition.

• Mathematical Grounding: Provides a way to integrate skills into standard RL algorithms like Q-learning.
• Automatic Skill Discovery: Algorithms can learn the initiation and termination conditions of options from data.
• Compositionality: Complex tasks can be solved by chaining pre-learned options. This framework is the theoretical backbone for most modern HRL algorithms, enabling rigorous analysis of hierarchical policies.

FeUdal Networks are a deep learning architecture for HRL. They consist of two modules: a Manager and a Worker.

• The Manager operates at a lower temporal resolution, setting abstract, goal-directed sub-goals in a latent state space.
• The Worker learns to execute primitive actions to achieve the Manager's current sub-goal.
• A key innovation is the goal embedding space, where sub-goals are directions, encouraging the Worker to learn directional skills. This separation of concerns allows the Manager to learn long-term strategy while the Worker masters short-term control, significantly improving performance on tasks requiring deep exploration like Montezuma's Revenge.

Hierarchical Actor-Critic is an algorithm designed for sparse reward environments. It builds a hierarchy of policies where each level learns to achieve the goals set by the level above it.

• Goal-Conditioned Policies: Each level's policy is trained to reach a specific sub-goal state.
• Internal Reward: A level receives a positive reward only when its goal is achieved, creating a clear credit assignment signal up the hierarchy.
• Backward Recursive Goal Setting: Higher levels propose goals that are feasible for the level below, learned through experience. HAC has been shown to solve tasks requiring thousands of primitive actions where flat RL algorithms fail entirely, demonstrating the power of hierarchical decomposition for exploration.

Temporal abstraction is the core principle enabling HRL, allowing an agent to operate over extended time horizons without detailed low-level planning for every step. It involves grouping sequences of primitive actions into reusable skills or options. For example, a robot's high-level policy might call the skill 'navigate to kitchen,' which itself executes hundreds of lower-level motor commands. This abstraction is formalized through the options framework, where an option is defined by a policy, a termination condition, and an initiation set.

The Options Framework is the predominant formalization for HRL, introducing the concept of a temporally extended action called an option. An option is a triple (I, π, β) where:

• I is the initiation set of states where the option can be started.
• π is the intra-option policy that selects primitive actions.
• β is the termination function giving the probability the option stops in each state. This framework allows the agent to learn a policy over options at a higher level, while the options themselves handle the lower-level control. It seamlessly integrates with standard RL algorithms like Q-learning, extended to SMDP (Semi-Markov Decision Process) Q-learning.

Feudal Reinforcement Learning is a specific HRL architecture inspired by managerial hierarchies. It features a strict top-down command structure where a manager agent sets high-level, abstract goals over a longer time scale, and a worker agent learns to achieve these goals through low-level actions. The manager provides goals, not actions, and the worker is rewarded for achieving them. This separation of concerns enforces a strong form of temporal abstraction and can improve exploration efficiency by constraining the worker's search space.

Skill Discovery refers to the autonomous learning of useful, reusable subtasks (skills or options) without explicit human specification. This is a major challenge in HRL. Common approaches include:

• Diversity-based Intrinsic Motivation: Encouraging the agent to discover skills that lead to diverse states or outcomes.
• Variational Autoencoders (VAEs): Learning a latent space of skills where different dimensions correspond to different behaviors.
• Graph-Based Methods: Clustering states visited in trajectories to identify natural subgoals. Successful skill discovery creates a library of primitives that a higher-level policy can then sequence to solve complex tasks efficiently.

MAXQ Value Function Decomposition is a hierarchical method that decomposes the overall value function Q(s, a) into a sum of value functions for individual subtasks. It breaks a task into a directed acyclic graph of subtasks. Each subtask has its own local reward function and local value function. The global Q-value is computed by summing the completion value of the current subtask with the values of all its parent subtasks. This decomposition provides a theoretical guarantee of optimality under the hierarchical policy and offers improved interpretability by showing the contribution of each subtask to the total reward.

In HRL, intrinsic motivation is often used to automatically generate subgoals, which are intermediate states that facilitate learning long-horizon tasks. Unlike external task rewards, intrinsic rewards are generated internally to guide exploration. Key methods include:

• Curiosity-Driven Exploration: Rewarding the agent for visiting novel or hard-to-predict states.
• Empowerment: Maximizing an agent's influence over its future states.
• Subgoal Testing: Proposing a candidate state as a subgoal and giving an intrinsic reward if reaching it improves performance on the main task. These self-generated subgoals create a natural hierarchy, where achieving a subgoal is a simpler task that contributes to the final objective.