REVERIE

Unlike navigation to a coordinate or room, REVERIE instructions specify a target object using abstract, human-like language (e.g., 'Bring me the book on the nightstand in the second bedroom'). The agent must interpret the referring expression, navigate to the correct room, and identify the specific object instance. This requires sophisticated visual grounding and spatial reasoning beyond simple path following.

The benchmark is built on the Matterport3D dataset, which consists of 3D reconstructions of real building interiors. This provides:

• Realistic visual textures and lighting.
• Complex, cluttered layouts mimicking real homes.
• Diverse object placements across different rooms. Training and evaluation in these environments is crucial for developing models that can transfer to real-world robotics applications, bridging the sim-to-real gap.

REVERIE decomposes the problem into two measurable sub-tasks:

1. Navigation: The agent must move from a start point to the vicinity of the target object.
2. Remote Grounding: Upon stopping, the agent must correctly identify the target object from a set of candidate objects visible in its final panorama. This structure allows for granular evaluation, isolating failures in wayfinding from failures in visual recognition and language grounding.

Performance is measured using a combination of navigation and grounding metrics:

• Navigation Success (NS): Percentage of episodes where the agent stops within a predefined distance (e.g., 3 meters) of the target object.
• Remote Grounding Success (RGS): Percentage of episodes where the agent both navigates successfully and correctly identifies the target object.
• Success weighted by Path Length (SPL): The primary metric, which penalizes successful but inefficient paths. It is calculated as: SPL = (1/N) * Σ (S_i * (L_i / max(P_i, L_i))), where S_i is success, L_i is the optimal path length, and P_i is the agent's path length.

The REVERIE dataset contains:

• 21,702 human-annotated instructions paired with trajectories in 90 Matterport3D buildings.
• An average instruction length of 18 words, describing objects and their locations relationally.
• 4,140 target objects spanning 1,000+ unique categories.
• A standard split for training, validation seen/unseen, and testing to evaluate generalization to novel environments and object configurations.

REVERIE builds upon and differs from related tasks:

• Vs. Vision-and-Language Navigation (VLN): VLN (e.g., R2R) requires navigating to a described location. REVERIE adds the critical layer of fine-grained object identification.
• Vs. Embodied Question Answering (EQA): EQA involves answering questions by exploring. REVERIE requires a specific physical action (navigating to an object) as the output.
• Vs. ALFRED: ALFRED focuses on interactive manipulation sequences. REVERIE is primarily a navigation and visual search task, though both use high-level language.

Vision-and-Language Navigation (VLN) is the foundational task of enabling an embodied agent to follow natural language instructions to navigate through a real or simulated 3D environment using visual perception. It is the core problem that REVERIE builds upon.

• Key Difference: VLN typically requires navigating to a described location (e.g., 'the kitchen'), while REVERIE requires navigating to and identifying a specific target object (e.g., 'the blue mug on the counter').
• Primary Inputs: Egocentric visual stream and a step-by-step navigation instruction.
• Common Benchmark: The Room-to-Room (R2R) dataset.

Embodied Instruction Following is the core problem in Embodied AI where an agent must execute a sequence of low-level actions in a physical or simulated environment to complete a task specified by a natural language instruction. REVERIE is a specific instantiation of this problem focused on remote object localization.

• Scope: Encompasses both navigation and object interaction tasks.
• Complexity: Instructions can be long-horizon and compositional (e.g., 'Go to the kitchen, pick up the mug, and place it in the sink').
• Exemplar Benchmark: ALFRED (Action Learning From Realistic Environments and Directives), which requires completing interactive tasks with object manipulation.

A Visual Referring Expression is a natural language phrase that uniquely identifies a target object or region within a visual scene. Comprehending these expressions is the central language understanding challenge in the REVERIE task.

• Purpose: To disambiguate a target from other similar objects using attributes (color, size), spatial relationships ('on the left'), and context ('the book you were reading').
• Key Ability: Requires visual grounding—linking linguistic concepts to specific pixels or 3D locations.
• Evaluation: Success is measured by the agent's ability to correctly select the referred-to object from all candidates in the scene.

Object Goal Navigation (ObjectNav) is the task of navigating to an instance of a specified object category (e.g., 'find a chair') in a previously unexplored environment using only egocentric visual input. It shares the 'find an object' goal with REVERIE but differs in key aspects.

• Instruction Simplicity: Uses a single object category label, not a complex referring expression.
• No Remote Specifier: The agent must explore to find any instance of the category, not a specific one described from a remote viewpoint.
• Core Challenge: Exploration, mapping, and semantic search without a prior map.

Success weighted by Path Length (SPL) is the primary quantitative metric for evaluating navigation performance in tasks like REVERIE and VLN. It rigorously balances success rate with efficiency.

• Calculation: SPL = (1/N) * Σ (S_i * (L_i / max(P_i, L_i))) where S_i is success (1/0), L_i is the optimal path length, and P_i is the agent's path length for instruction i.
• Interpretation: A score of 1.0 means the agent succeeded on all episodes via the shortest possible path. A longer, meandering path reduces the score even for successful episodes.
• Importance: Prevents agents from learning degenerate policies that succeed by exhaustive exploration.

A Language-Conditioned Policy is a neural network controller that outputs actions (e.g., move forward, turn left, stop) based on both the current visual observation and an embedded natural language instruction. This is the core algorithmic component of a REVERIE agent.

• Architecture: Typically uses a Cross-Modal Transformer to fuse visual features from a CNN and linguistic features from a language encoder (like BERT).
• Training: Can be trained via Behavior Cloning (imitation learning) on expert trajectories, or Reinforcement Learning to maximize task success.
• Output: May predict low-level actions directly or intermediate waypoints for a planner to follow.