ALFRED

ALFRED tasks are composite, requiring agents to execute sequences of 50-100 low-level actions to achieve a high-level goal. Each task is decomposed into multiple sub-goals.

• Example Task: 'Put a cooled apple slice on the counter.' This requires the agent to: 1) Find an apple, 2) Pick it up, 3) Find a knife, 4) Slice the apple, 5) Find a fridge, 6) Cool the slice, 7) Place it on the counter.
• This structure tests sequential decision-making and planning over extended timeframes, moving beyond simple single-step commands.

Tasks are specified through two complementary modalities that the agent must fuse:

• Natural Language Instruction: A high-level goal described in English (e.g., 'Clean all the mugs on the table').
• Visual Demonstration: A third-person video of a human expert trajectory completing the same task, providing disambiguating visual context.
• The agent must ground the language in the visual demonstration and its own egocentric view to understand actionable object states and spatial relationships.

Unlike pure navigation benchmarks, ALFRED requires physical interaction with a diverse set of objects using a defined action space.

• Action Primitives: The agent executes actions like Pickup, Put, Slice, Toggle, Heat, Clean, and Navigate.
• Object State Changes: Success depends on altering object states (e.g., a tomato must be sliced, then cooked). The agent must perceive and reason about these state transitions.
• This focuses evaluation on embodied interaction, not just perception or navigation.

ALFRED employs a multi-tiered evaluation protocol to diagnose agent capabilities and failures.

• Primary Metric: Task Success Rate - Did the agent complete the entire high-level instruction?
• Goal-Conditioned Success: Measures success for each sub-goal within a task.
• Path-Weighted Metrics: Evaluate efficiency, similar to Success weighted by Path Length (SPL), but adapted for interactive tasks.
• This granular evaluation helps pinpoint whether failures occur in language understanding, visual grounding, planning, or low-level control.

The benchmark is designed to stress-test an agent's ability to ground language in perception and action and to generalize.

• Seen vs. Unseen Splits: Environments and objects are divided into training (seen) and validation/test (unseen) sets.
• Language Variation: Instructions for the same task use diverse phrasings.
• Partial Observability: The agent operates from an egocentric view without a map, requiring it to maintain and update an internal world model.
• Success requires moving beyond memorization to true compositional understanding of language and environment dynamics.

This task requires the agent to navigate to a target object, pick it up, and place it in a specified location, often involving intermediate steps like clearing obstacles. For example, an instruction might be: "Put the cooled apple on the counter. First, pick up the apple from the fridge and move the vase blocking the counter."

• Key Challenge: Long-horizon planning with sequential dependencies.
• Manipulation Primitives: Pickup, Put, Toggle (to open/close containers).
• Visual Grounding: Must distinguish between similar objects (e.g., a hot apple vs. a cooled apple).

This task tests the agent's ability to understand object states and use tools. The agent must find an object, pick it up, and then examine it under a light source (e.g., a lamp or a flashlight). An example instruction: "Look at the vase under the lamp."

• Key Challenge: Understanding the functional purpose of objects (a lamp provides light).
• State Change: The agent must often first Toggle a light source to the 'on' state.
• Multi-Modal Reasoning: Combines object recognition with understanding affordances (what actions an object enables).

This involves tidying or organizing multiple objects in a scene. Tasks often require moving objects to their proper receptacles or arranging them in a specific order. Example: "Clean the table by putting the dirty mug in the sink and the book on the shelf."

• Key Challenge: Multi-object interaction and task decomposition.
• Semantic Knowledge: Requires implicit understanding of object categories (dirty dishes go in sink, books belong on shelves).
• Partial Observability: The agent may need to explore to find all target objects and their destinations.

This task requires the agent to change the temperature state of an object using appliances. The agent must find an object, place it in a heating (microwave) or cooling (fridge) appliance, operate the appliance, and then retrieve the object. Example: "Heat the potato in the microwave."

• Key Challenge: Understanding object state transitions and appliance operation sequences.
• Action Sequence: Navigate → Pickup → Put (into appliance) → Toggle (appliance on) → Wait → Toggle (appliance off) → Pickup → Place.
• Temporal Reasoning: The agent must recognize when the state change is complete.

These are the most complex tasks in ALFRED, combining multiple primary task types into a single long-horizon instruction. They test an agent's ability to decompose instructions, maintain memory of subgoals, and re-plan based on environment state. Example: "Make a simple breakfast. Put the bread in the toaster and pour a glass of orange juice."

• Key Challenge: Hierarchical planning and robust execution over 50+ low-level actions.
• Instruction Decomposition: The agent must infer the sequence: find bread → put in toaster → operate toaster → find juice → find glass → pour juice.
• Common-Sense Reasoning: Requires world knowledge (e.g., toast is made from bread, juice is poured into a glass).

Each ALFRED task is defined by a high-level goal, a step-by-step natural language instruction, and a visual demonstration of an expert completing the task. The data is structured as Trajectory-Instruction Pairs.

• Expert Demonstration: A sequence of low-level actions (e.g., MoveAhead, RotateRight, Pickup) and corresponding egocentric view RGB frames.
• Language Annotations: Includes both a high-level goal description ("Put a hot apple on the counter") and a detailed step-by-step list ("1. Go to the fridge...").
• Evaluation: Uses metrics like Task Success and Goal-Condition Success, which measure if each subgoal and the final goal were achieved.

The foundational task of navigating a 3D environment based on a natural language instruction using visual perception. ALFRED extends VLN by requiring object interaction and long-horizon task completion, not just point-to-point navigation.

• Core Difference: VLN agents typically stop upon reaching a location; ALFRED agents must manipulate objects to alter the environment's state.
• Benchmark Example: The Room-to-Room (R2R) dataset is a canonical VLN benchmark for navigation-only instructions.

The overarching problem domain where an agent executes low-level actions (e.g., MoveAhead, Pickup, Toggle) in a simulated or physical environment to complete a task specified by language. ALFRED is a specific, complex instantiation of this problem.

• Action Space: Defined as a set of atomic motor commands.
• Goal: To learn a language-conditioned policy that maps visual observations and instructions to action sequences.

A benchmark for Remote Embodied Visual Referring Expression in Real Indoor Environments. Like ALFRED, it requires object interaction, but the instruction is a high-level referring expression (e.g., 'bring me the blue mug on the nightstand') rather than a step-by-step guide.

• Key Challenge: Requires visual grounding to identify the target object from a vague description, followed by navigation and fetching.
• Contrast with ALFRED: REVERIE tests spatial reasoning and reference resolution; ALFRED tests sequential instruction parsing and execution.

A critical cognitive capability for solving ALFRED tasks. It involves breaking a long, compound instruction (e.g., 'make a cooled coffee') into a sequence of executable sub-goals (find kettle, fill with water, find coffee, etc.).

• Architectural Approach: Often handled by a planner module or a large language model that generates a step-by-step plan.
• Failure Mode: A primary source of error in ALFRED is the agent losing track of which sub-task it is executing, leading to irrecoverable action sequences.

The primary evaluation metric for navigation-heavy embodied tasks, including ALFRED. It measures success rate while penalizing inefficient paths.

• Formula: SPL = (1 / N) * Σ (S_i * (L_i / max(P_i, L_i))) where S_i is success (0/1), L_i is optimal path length, and P_i is agent's path length.
• Interpretation: An SPL of 1.0 means the agent succeeded via the shortest possible path every time. ALFRED reports separate Task Success and Goal-Condition Success, with SPL applied to the navigation components.

The two primary simulation platforms used for embodied AI research and benchmarking. ALFRED is built on AI2-THOR.

• AI2-THOR: Provides physics-enabled, interactive indoor environments. Critical for ALFRED's object manipulation tasks (e.g., slicing, toggling).
• Habitat: Focuses on high-performance, photorealistic simulation for efficient training, often used for large-scale navigation.
• Role: These simulators provide the environment API, action space, and visual renderer necessary to train and evaluate agents like those for ALFRED without physical robots.