Options Framework

The initiation set defines the set of states from which an option can be started. It is a subset of the environment's state space, often denoted as I ⊆ S. This component determines when a temporally extended skill is available for execution.

• Formal Role: Acts as a precondition or gating function.
• Example: For a robot's 'open door' option, the initiation set includes states where the robot is near a closed door and its gripper is empty.
• Key Property: If the agent is not in a state within the initiation set, the option cannot be invoked, ensuring actions are contextually appropriate.

The internal policy (π) is the strategy that dictates which primitive action to take at each step while the option is executing. It maps from states within the option's scope to primitive actions (a ∈ A).

• Formal Role: The 'how' of the option—the sequence of low-level controls.
• Example: For a 'navigate to kitchen' option, the internal policy would be a mapping from sensor inputs (lidar, camera) to wheel velocities.
• Key Property: This policy is followed until the termination condition is met, abstracting away a potentially long sequence of primitive actions into a single decision point.

The termination condition (β) is a function that specifies the probability of the option terminating in any given state. It defines when the temporally extended action is considered complete, returning control to a higher-level policy.

• Formal Role: β(s) → [0,1], where β(s) is the probability of terminating upon entering state s.
• Example: For the 'pick up cup' option, β(s) = 1.0 when the cup is securely in the gripper, and 0.0 otherwise.
• Key Property: Provides temporal abstraction; the agent commits to the option's internal policy until β indicates termination, enabling planning over longer time horizons.

A critical feature of the options framework is that an option, once initiated, executes until termination independent of the higher-level policy that selected it. This is known as the semi-Markov property.

• Formal Implication: The future after an option is chosen depends only on the current state, not the history of states and actions within the option.
• Benefit: This property allows options to be treated as single, indivisible actions at a higher level of abstraction, enabling the use of standard MDP solution methods (like value iteration) over the new action set of options.

Options are the foundational building block for Hierarchical Reinforcement Learning (HRL). By grouping primitive actions, they create a hierarchy of policies.

• Two-Tier Learning: A meta-policy (or manager) learns to select among options, while each option's internal policy (or controller) learns the low-level skill.
• Skill Reuse: Learned options can be transferred across different but related tasks, dramatically improving sample efficiency.
• Planning Abstraction: In algorithms like Option-Critic, policies over options and intra-option policies can be learned end-to-end via gradient descent.

Consider a mobile robot in a warehouse. Its primitive actions are small wheel movements. Using the options framework, we can define:

• Option 1: 'Traverse Corridor'

  • Initiation Set: States where robot is at a corridor entrance.
  • Internal Policy: Follow the center line using visual servoing.
  • Termination Condition: β(s)=1.0 when the robot's GPS coordinates match the corridor's end.

• Option 2: 'Avoid Dynamic Obstacle'

  • Initiation Set: States where a moving person is detected within 2 meters.
  • Internal Policy: A reactive policy to steer away and stop.
  • Termination Condition: β(s)=1.0 when the path is clear again.

A high-level planner can now sequence 'Traverse Corridor' and 'Avoid Dynamic Obstacle' as single actions, making long-horizon navigation tractable.

A mobile robot uses options to decompose a high-level navigation task. Instead of learning a single policy from pixels to motor torques, it learns reusable sub-policies (options).

• Navigate to Kitchen: Initiation set is any location; internal policy uses visual SLAM; termination condition is proximity to the counter.
• Open Door: Initiation set is near a door; internal policy controls the gripper; termination condition is door angle > 45 degrees. This abstraction allows the high-level planner to sequence [Navigate to Kitchen, Open Door, Navigate to Counter], reusing skills across different missions.

In complex environments like Montezuma's Revenge, options can be discovered automatically to represent meaningful skills.

• Option Discovery: Algorithms like Option-Critic or DIAYN (Diversity is All You Need) learn options that correspond to distinct behaviors (e.g., 'climb a ladder', 'collect a key') without explicit human labeling.
• Temporal Abstraction: The agent learns an option's internal policy and termination condition. The high-level policy then selects among these discovered options, drastically improving exploration and sample efficiency by committing to coherent sequences of primitive actions.

The Options Framework formally casts hierarchical RL as a Semi-Markov Decision Process (SMDP). An SMDP generalizes an MDP by allowing actions (now options) to take a variable amount of time.

• Key Formalism: The value function for an option is defined over states where the option can be initiated. The Bellman equation is extended to account for the cumulative reward and time elapsed during the option's execution.
• Planning Benefit: This allows the use of standard RL algorithms (like Q-learning) at the option level, treating each option as a single, temporally extended action in a new, abstracted SMDP.

A foundational end-to-end gradient-based method for simultaneously learning options and the policy over them.

• Two-Level Gradient: The architecture computes gradients for both the intra-option policies (how to execute the option) and the termination functions (when to stop the option), all in service of maximizing the overall task reward.
• On-Policy Learning: It learns options specialized for the current task without pre-defining subgoals. This demonstrates how useful temporal abstractions can emerge directly from the reward signal, validating the framework's utility for autonomous skill formation.

Options provide natural subgoals for planning algorithms like Monte Carlo Tree Search (MCTS) or Model-Predictive Control (MPC).

• Abstract Search Tree: Instead of expanding a search tree with primitive actions at every node, the planner expands with options. Each node now represents executing a skill for multiple timesteps.
• Efficient Planning: This dramatically reduces the depth and breadth of the search tree, enabling longer-horizon reasoning. In a learned world model, rolling out option sequences provides more coherent and informative simulated trajectories than rolling out random primitive actions.

Options offer a structured way to incorporate expert demonstrations into RL.

• Expert Segments as Options: Demonstrated trajectories can be segmented into chunks that define option policies. The initiation set is the state where the expert began the segment, and termination is the state where they ended it.
• Hybrid Learning: The agent can use these expert-derived options as a robust starting point. Its high-level policy learns to sequence them, and its low-level policies can refine the option execution via RL. This is particularly powerful in robotic manipulation, where safe, sensible skills are bootstrapped from demonstration.

A Semi-Markov Decision Process (SMDP) is the mathematical model underlying the Options Framework. It generalizes a standard MDP by allowing actions (now called options) to take a variable amount of time to complete.

• Formalization: An option's execution is a single, temporally extended action within the SMDP.
• Components: The SMDP defines a reward and transition probability for the entire duration of an option, not just a single timestep.
• Solution Methods: Value functions and optimality equations can be defined over options, enabling planning and learning at the abstract level.

The initiation set is a core component of an option, defining the set of states from which the option can be invoked. It enforces the precondition for using a temporally extended skill.

• Function: Determines where a specific skill or subroutine is applicable.
• Example: A pick-up-cup option might only be initiated in states where a cup is within the agent's reach.
• Learning: Can be learned from data or specified by a designer. A poorly defined initiation set can lead to the agent attempting skills in invalid states.

The termination condition of an option is a function that determines, at each state, the probability that the option terminates. It defines the postcondition or stopping criterion for the extended action.

• Role: Signals when control should be returned to a higher-level policy to select a new option.
• Types: Can be deterministic (e.g., terminate upon reaching a goal region) or stochastic.
• Critical for Composition: Well-defined termination conditions allow options to be reliably sequenced to accomplish complex tasks.

The intra-option policy is the low-level, primitive-action policy that is executed while an option is active. It maps states encountered during the option's execution to primitive actions.

• Scope: Operates only from initiation until termination of its option.
• Learning: Can be learned using any standard RL algorithm, with the option providing the task context.
• Abstraction: This policy is hidden from the higher-level agent, which only sees the option as a single, abstract choice.