Hierarchical Reinforcement Learning (HRL)

The core mechanism of HRL that allows agents to reason over extended time horizons by grouping primitive actions into reusable subroutines called options or skills. Instead of planning every micro-action, a high-level policy selects which macro-action to execute, dramatically reducing the decision horizon. In logistics, this means a meta-controller might choose a 'cross-dock' option while a low-level controller handles the specific conveyor routing.

A formal mathematical model for HRL where an option is defined by three components:

• Initiation Set: The states where the option can begin
• Policy: The low-level actions to execute
• Termination Condition: The probability of ending the option in each state This formalization allows options to be composed, learned, and transferred across related tasks, such as reusing a 'palletize' option across different warehouse layouts.

A paradigm where a high-level controller periodically sets subgoals for a lower-level controller to achieve. The high-level policy operates at a coarser timescale, specifying target states like 'move to zone B-12', while the low-level policy learns to reach arbitrary subgoals. This decoupling enables zero-shot generalization—a low-level navigation policy trained to reach any warehouse coordinate can be immediately reused by new high-level strategies.

An early and influential HRL architecture inspired by feudal systems. A Manager module sets latent goals for a Worker module, which learns to maximize intrinsic reward based on goal achievement rather than the extrinsic environment reward. The Manager learns to set goals that maximize external reward. This creates a natural information hiding principle where the Manager does not need to know how the Worker accomplishes its directives, simplifying coordination in multi-agent logistics fleets.

HRL agents can autonomously discover reusable skills without human decomposition by maximizing mutual information between skills and the states they reach. Algorithms like DIAYN (Diversity Is All You Need) train a discriminator to identify which skill is being executed, rewarding the agent for making its behavior predictable and distinct. In supply chain contexts, this can autonomously discover skills like 'consolidate shipments' or 'reroute to alternate DC' without manual engineering.

HRL naturally maps to multi-echelon supply chain structures. A top-level policy makes strategic decisions (e.g., which regional DC to source from), a mid-level policy handles tactical routing between hubs, and a low-level policy manages last-mile delivery execution. Each level operates on a different clock—strategic decisions update weekly, tactical daily, and operational in real-time—mirroring the natural hierarchy of enterprise planning and enabling graceful degradation when lower-level policies are retrained independently.

The core mechanism enabling HRL agents to reason over extended time horizons rather than primitive actions. By grouping sequences of primitive actions into options or skills, the agent compresses decision-making into higher-level choices.

• Options Framework: A formal model where each option consists of an initiation set, a sub-policy, and a termination condition
• Semi-Markov Decision Processes (SMDPs): The mathematical foundation that extends MDPs to handle variable-duration actions
• Enables planning at the level of "deliver to region" rather than individual steering and acceleration commands

A pioneering HRL architecture where managers set abstract goals for workers who learn to achieve them independently. This feudal hierarchy creates a natural division of labor:

• Manager Level: Sets latent goals or reward signals for subordinate workers
• Worker Level: Learns sub-policies to satisfy manager directives using primitive actions
• Goal Embedding: Managers communicate via learned vector representations rather than explicit instructions

Applied to supply chains, a strategic manager might set a "minimize regional inventory variance" goal while warehouse workers execute local fulfillment policies.

A formalism where the agent's policy is constrained by a finite state machine with internal memory, restricting the policy to execute only within predefined hierarchical programs.

• Machine States: Can be action states (execute primitive), call states (invoke sub-machine), or choice states (non-deterministic selection)
• Partial Policies: HAMs define a restricted policy space, dramatically reducing the search complexity
• Learning: Occurs only at choice states where the agent selects among available transitions

This approach is particularly valuable for logistics workflows with regulatory or safety constraints that must never be violated during exploration.

A mathematical framework that decomposes the overall value function into a hierarchical sum of subtask value functions, enabling independent learning at each level.

• Value Function Decomposition: V(parent, state) = V(subtask, state) + Completion Function
• Subtask Independence: Each subtask learns its own locally optimal policy without needing global context
• State Abstraction: Subtasks can operate on reduced state representations, ignoring irrelevant variables

In fleet management, a high-level dispatcher learns routing values while low-level controllers independently learn fuel-efficient driving, with MAXQ ensuring mathematical consistency.

Methods for automatically discovering useful hierarchical decompositions without manual specification, critical for scaling HRL to complex logistics environments.

• Diversity is All You Need (DIAYN): Learns diverse skills by maximizing mutual information between skills and visited states
• Variational Option Discovery: Uses variational inference to identify reusable behavioral primitives from experience
• Betweenness Centrality: Identifies bottleneck states in the environment's transition graph as natural subgoals

These techniques enable autonomous discovery of logistics primitives like "consolidate pallets" or "rebalance zone" from raw operational data.