Instruction Decomposition

Instruction decomposition transforms a high-level, abstract goal into a hierarchical plan of concrete, low-level actions. This is essential because a single instruction like 'Bring me the coffee mug from the desk in the study' involves multiple implicit steps: navigating to the study, identifying the desk, locating the mug, picking it up, and returning. The process reduces planning horizon complexity by creating manageable sub-problems, each with a clear success condition.

• Example: The instruction 'Go to the kitchen and put the kettle on the stove' decomposes into: 1) Navigate to kitchen doorway, 2) Locate kettle, 3) Pick up kettle, 4) Locate stove, 5) Place kettle on stove burner.

The decomposition process must parse spatial relationships ('next to', 'inside', 'on the left') and temporal or causal order ('first', 'then', 'after that') embedded in the instruction. This involves linguistic analysis to extract landmarks, directional cues, and sequential constraints. A model must understand that 'before entering the bedroom, turn off the hallway light' specifies an order of operations dependent on the agent's location relative to two distinct areas.

Failure to correctly parse these elements leads to execution errors, such as attempting to interact with an object before reaching the correct room or violating a specified sequence.

Natural language instructions are often ambiguous. Decomposition acts as a disambiguation mechanism by leveraging environmental context. The phrase 'go to the white chair' is ambiguous if multiple white chairs exist. A decomposition system must generate sub-goals that include an exploration phase or use visual grounding to resolve the reference.

This often involves:

• Active Perception: Sub-goals to move to a vantage point for better observation.
• Iterative Re-grounding: Continuously matching instruction phrases to the evolving perceptual stream.
• Implicit Constraint Inference: Understanding that 'my coffee mug' likely refers to a mug in a frequently visited location like a home office.

Decomposition creates a bridge between symbolic AI planning and neural policy execution. The high-level sequence can be treated as a symbolic plan where each sub-goal (e.g., At(Kitchen), Holding(Kettle)) is a state to achieve. A lower-level language-conditioned policy or visuomotor controller then executes the actions to reach each sub-goal.

Architectural Patterns:

• Module Networks: Use a dedicated language module for decomposition and a separate vision/action module for execution.
• End-to-End Learning: Train a single model (e.g., a Cross-Modal Transformer) to implicitly perform decomposition within its forward pass through attention mechanisms.
• Hybrid Systems: Use a learned decomposer to generate sub-goals, which are then fulfilled by classical Task and Motion Planning (TAMP) algorithms.

A robust decomposition method enables compositional generalization—the ability to understand and follow novel combinations of known concepts. If an agent can decompose 'pick up the apple' and 'go to the kitchen', it should, in principle, be able to handle the novel instruction 'go to the kitchen and pick up the apple'. This relies on learning a library of reusable action primitives and spatial concepts.

Evaluation of this capability is done through zero-shot navigation tests on unseen environment layouts or instruction phrasing. Benchmarks like ALFRED and REVERIE explicitly test an agent's ability to decompose long-horizon, interactive instructions it has never seen during training.

Decomposition is not a one-time, feed-forward process. Effective embodied agents employ closed-loop decomposition, where the sequence of sub-goals is continually verified and adjusted based on execution feedback. If a sub-goal fails (e.g., the target object is not found), the system must re-decompose the remaining instruction from the new state or formulate a recovery sub-goal.

This introduces a recursive structure similar to hierarchical reinforcement learning, where high-level actions (sub-goals) can be expanded into finer-grained sequences upon failure. This capability is critical for robustness in real-world environments where perceptual aliasing and dynamic changes are common.

High-fidelity simulators provide the training and testing grounds for instruction decomposition research.

• Habitat: A high-performance platform supporting photorealistic 3D environments from datasets like Matterport3D. It enables fast training of language-conditioned policies for navigation.
• AI2-THOR: Features fully interactive, physics-enabled indoor environments. It is essential for benchmarks like ALFRED, where decomposition requires predicting actions that change object states (e.g., turning on a faucet).

These tools allow for the scalable generation of trajectory-instruction pairs and rigorous evaluation.

Performance on instruction decomposition is measured by metrics that balance success, efficiency, and grounding accuracy.

• Success weighted by Path Length (SPL): The primary metric for navigation. It is defined as (Success * Optimal Path Length) / max(Agent Path Length, Optimal Path Length). It penalizes long, inefficient paths.
• Task Success Rate: The fraction of episodes where the agent completes all sub-goals correctly (e.g., in ALFRED).
• Goal Condition Success (GCS): Used in REVERIE, measures if the agent correctly identifies the target object specified in the instruction.

These metrics ensure agents are evaluated on both the correctness of their decomposition and the quality of execution.

Vision-and-Language Navigation (VLN) is the core benchmark task where an embodied agent must follow a natural language instruction to navigate through a real or simulated 3D environment using egocentric visual perception. It is the primary application domain for instruction decomposition.

• Key Challenge: The agent must perform cross-modal alignment between the instruction's semantics and the visual scene to make sequential navigation decisions.
• Standard Datasets: Benchmarks like Room-to-Room (R2R) and REVERIE provide trajectory-instruction pairs for training and evaluation.

Embodied Instruction Following is a broader problem that extends VLN to include object interaction. An agent must execute a long-horizon sequence of low-level actions (e.g., move forward, pick up) in a physical or simulated environment to complete a task specified by a natural language instruction.

• Complexity: Requires decomposing instructions into both navigation and manipulation sub-goals.
• Benchmark Example: ALFRED is a prominent benchmark that requires agents to complete interactive household tasks like "put a cooled apple on the table," which necessitates multi-step planning and action primitive execution.

Instruction Grounding is the perceptual process by which an agent maps the semantic concepts and spatial relations (e.g., "to the left of the blue chair") described in a natural language instruction to specific visual observations and actionable locations in its environment. It is the prerequisite step that makes decomposition possible.

• Core Mechanism: Often achieved via cross-modal transformers that attend over visual features and language tokens.
• Output: Produces a visual referring expression that pinpoints the target object or region for the next sub-goal.

Task and Motion Planning (TAMP) is the classical robotics hierarchy that Instruction Decomposition often mimics or learns. It involves a high-level task planner that outputs a sequence of symbolic actions, which are then fed to a low-level motion planner that generates feasible trajectories.

• Relation to Decomposition: In learned systems, the language-conditioned policy performs the role of the task planner, outputting waypoint predictions or action primitives.
• State Representation: Effective TAMP relies on a robust world model or semantic map to reason about action feasibility.

A Language-Conditioned Policy is the learned neural network controller that implements instruction decomposition. It is a function that outputs actions (e.g., motor commands) based on the current egocentric view and an embedded natural language instruction.

• Architecture: Often built using a cross-modal transformer to fuse visual and linguistic features.
• Training: Can be trained via behavior cloning for navigation on expert demonstrations or via reinforcement learning.
• Deployment: A major challenge is sim-to-real transfer to deploy policies trained in simulators like Habitat or AI2-THOR to physical robots.

Success weighted by Path Length (SPL) is the primary quantitative metric for evaluating instruction-following navigation agents. It measures the success rate while penalizing inefficient paths.

• Calculation: SPL = (1/N) Σ (S_i * (L_i / max(P_i, L_i))) where S_i is success (0/1), L_i is the optimal path length, and P_i is the agent's path length for instruction i.
• Purpose: Directly evaluates the efficiency of an agent's decomposition and planning. An agent that reaches the goal via a meandering path receives a lower SPL score than one that takes a near-optimal route.
• Standardization: This metric is used across all major VLN benchmarks (R2R, REVERIE, ALFRED).