Hierarchical RL (HRL)

The Options Framework formalizes temporally extended actions. An option is a triple (I, π, β) where:

• I is the initiation set of states where the option can be started.
• π is the intra-option policy that selects primitive actions.
• β is the termination function giving the probability the option stops in a given state. The agent operates in a Semi-Markov Decision Process (SMDP), where actions (options) take variable amounts of time. Learning occurs via intra-option learning (updating the policy π) and option-level learning (treating options as actions in the SMDP).

Feudal Networks implement a managerial hierarchy. A manager network operates at a lower temporal resolution, setting abstract goals (or directives) in a latent goal space. A worker network receives these goals and learns to output primitive actions that achieve them over a fixed time interval. The manager is trained to propose goals that maximize the extrinsic reward, while the worker is trained with an intrinsic reward based on how well its achieved state matches the manager's goal. This separation of concerns enforces sub-task decomposition.

Hierarchical Actor-Critic (HAC) is a framework for goal-conditioned HRL with hindsight experience replay. It features multiple levels of goal-conditioned policies.

• Each level's actor proposes a sub-goal for the level below it.
• The lowest level executes primitive actions.
• A sub-goal test function determines if a proposed sub-goal is achievable by the level below.
• Hindsight Action Transitions are created when a sub-goal is not met, treating the achieved state as a virtual goal for training, drastically improving sample efficiency in sparse reward settings.

MAXQ decomposes the value function of the overall task into a sum of value functions for individual subtasks. It defines a task graph where child nodes are subtasks of parent nodes. The value of a parent task Q(p, s, a) is decomposed as: Q(p, s, a) = V(a, s) + C(p, s, a) where V(a, s) is the completion value of child task a in state s, and C(p, s, a) is the completion function representing the expected cumulative reward after child task a finishes until parent task p finishes. This enables hierarchical policy evaluation and off-policy learning across subtasks.

A major challenge in HRL is automatically discovering useful skills (options). Unsupervised skill discovery methods aim to learn a diverse set of skills without a task-specific reward.

• DIAYN (Diversity is All You Need): Maximizes the mutual information between states and skills I(S; Z), encouraging skills to be distinguishable by the states they visit.
• Variational Intrinsic Control: A similar information-theoretic objective that learns skills leading to different outcomes. These discovered skills form a primitives library that can be rapidly composed by a high-level controller to solve new tasks, enabling zero-shot or few-shot adaptation.

Hierarchies of Abstract Machines (HAMs) model hierarchy through finite-state machines. A HAM is a partial program that constrains the agent's choices, reducing the effective size of the action space. The learning problem becomes finding the best policy within the constraints of the machine. This differs from options, as the hierarchy is defined by a program structure (states, transitions, choices) rather than initiation/termination functions. It provides a strong inductive bias, making learning more tractable when the task structure is partially known, and bridges programming logic with learned control.

The Options Framework is a formalization of temporal abstraction in HRL, where an option is a triple (I, π, β). I is the initiation set (states where the option can start), π is the intra-option policy (how to act), and β is the termination condition (probability of stopping in a state). This framework allows the agent to operate over extended time steps using macro-actions, treating the execution of an option as a single, high-level action in a semi-Markov Decision Process (SMDP). It is the primary mathematical model for implementing skill hierarchies.

Feudal Reinforcement Learning is a specific HRL architecture inspired by managerial hierarchies. A manager module operates at a high level and sets abstract, long-term goals for a worker module. The worker learns to achieve these goals through low-level actions. The manager's goals are defined in a different, often more abstract, state space than the worker's observations. This separation of concerns facilitates credit assignment across long time horizons and enables the transfer of managerial policies to new, similar tasks by reusing the same worker skills.

Skill Discovery refers to the unsupervised or intrinsic motivation-driven process of learning a repertoire of reusable low-level behaviors or skills before they are organized into a hierarchy. Methods include:

• Diversity-Driven: Maximizing the diversity of discovered skills (e.g., via mutual information).
• Goal-Conditioned: Learning policies that reach a set of discovered or random goals.
• Information Bottleneck: Learning skills that compress sensory input while preserving control. These discovered skills become the primitive actions for a higher-level HRL controller, enabling efficient hierarchical planning without manual specification of sub-tasks.

MAXQ Value Function Decomposition is a method for decomposing the overall value function of a task into the sum of value functions for individual subtasks. It defines a hierarchical value function Q(s, a) where a can be either a primitive action or a composite subtask. The value of a composite task is decomposed into the value of its child tasks plus a completion function. This decomposition enables:

• Policy reuse: Subtask policies can be shared across multiple parent tasks.
• State abstraction: Subtasks may ignore irrelevant parts of the state.
• Theoretical guarantees: Provides a framework for proving hierarchical optimality.

Hierarchical Actor-Critic architectures extend the standard Actor-Critic framework to multiple levels of hierarchy. Typically, a meta-controller (high-level critic) learns the value of choosing subtasks or goals, while a sub-controller (low-level actor) learns the policy to achieve them. Each level may have its own separate value function and policy network. Training often involves off-policy methods with experience replay, where transitions are tagged with the active high-level goal. This structure is central to modern deep HRL algorithms like HIRO (Data-Efficient Hierarchical Reinforcement Learning) and HAC (Hierarchical Actor-Critic).

Goal-Conditioned Hierarchical RL (GCHRL) frames HRL as learning to achieve arbitrary goals specified in a goal space. The hierarchy consists of:

• A high-level policy that selects intermediate sub-goals in the goal space at a low frequency.
• A low-level, goal-conditioned policy that learns to reach any given sub-goal from the current state. This decoupling allows the high-level policy to plan in an abstract, compact goal space, while the low-level policy acts as a general-purpose goal-reaching skill. The approach is highly amenable to transfer learning, as the low-level skills can be reused for new high-level tasks without retraining.