Habitat

The core 3D simulation engine provides photorealistic rendering and physics-based interaction. It is built for speed, supporting >10,000 simulation steps per second on a single GPU, which is critical for efficient reinforcement learning. The simulator loads 3D scene datasets (like Matterport3D and Gibson) and supports sensor configuration (RGB-D cameras, semantic segmentation), agent embodiment, and rigid-body dynamics for object interaction.

A Python library that defines the agent-environment interface and standard benchmark tasks. It provides:

• Standardized Gym-like API for agent stepping and observation.
• Pre-defined task definitions (e.g., PointNav, ObjectNav, Rearrangement).
• Dataset abstraction to load episodes—specific start states, goals, and instructions.
• Metrics calculation like Success weighted by Path Length (SPL). This abstraction allows researchers to focus on agent design rather than simulator integration.

Habitat's realism stems from its use of large-scale, scanned real-world environments. Key datasets include:

• Matterport3D: ~90 large-scale residential buildings with high-resolution textures.
• Gibson: 572 full buildings with semantically annotated 3D reconstructions.
• Replica: High-fidelity, artist-created reconstructions with perfect geometry. These datasets provide the geometric and semantic complexity necessary for training agents that can generalize to real-world spaces.

Habitat hosts standardized benchmarks to measure progress in Embodied AI. These include:

• PointNav: Navigate to a coordinate.
• ObjectNav: Find an object instance.
• ImageNav: Navigate to a location matching a target image.
• Rearrangement: Pick and place objects to restore a scene.
• Language-guided tasks like Vision-and-Language Navigation (VLN) via integration with datasets like R2R. Benchmarks provide leaderboards, evaluation servers, and baseline implementations for fair comparison.

Habitat is the de facto simulation backend for major Vision-and-Language Navigation (VLN) benchmarks. It provides the 3D environments and physics where agents learn to follow natural language instructions (e.g., 'Walk past the kitchen and stop at the blue chair in the living room').

• Core Datasets: Hosts and runs evaluations for Room-to-Room (R2R), REVERIE, and Touchdown.
• Metric Standardization: Precisely computes metrics like Success weighted by Path Length (SPL) and Navigation Error (NE).
• Fast Simulation: Enables orders-of-magnitude faster training than real-time, crucial for iterative RL and imitation learning research.

A core benchmark for exploration-based robotics. In Object Goal Navigation, an agent is spawned in an unseen environment and must find an instance of a target object category (e.g., 'find a toilet').

• No Instructions, Only Goals: Tests pure visual search and mapping capabilities.
• Semantic Understanding: The agent must recognize object categories from egocentric views.
• Habitat ObjectNav Challenge: A standardized benchmark with defined evaluation protocols using the Gibson and Matterport3D datasets.

Habitat 2.0 and later support interactive rearrangement, a complex task requiring both navigation and manipulation. The agent must restore a scene to a desired state, moving objects to target positions.

• Pick and Place: Involves physics-enabled interaction with objects.
• Benchmarks: Habitat Rearrangement Challenge tests both Pick and Place skills in disorganized environments.
• Realism: Uses realistically modeled object physics (mass, friction) and articulated objects (drawers, doors).

AI2-THOR (The House Of inteRactions) is a complementary simulation framework focused on interactive, physics-enabled environments. While Habitat emphasizes high-performance, photorealistic simulation for scalable navigation, AI2-THOR specializes in modeling object interactions (e.g., opening doors, picking up items) with precise physics. It is a key platform for benchmarks like ALFRED that require complex, multi-step task completion.

Matterport3D is a large-scale dataset of 3D reconstructions of real-world building interiors. It provides the foundational, photorealistic 3D environments used by the Habitat simulator for training and evaluation. The dataset includes:

• RGB-D panoramas for visual input.
• Mesh reconstructions for accurate physics and navigation.
• Semantic annotations for object categories. It is the primary environment source for benchmarks like Room-to-Room (R2R) and REVERIE.

The Gibson Environment is another photorealistic simulation platform for Embodied AI, built from 3D scans of real buildings. A key differentiator is its focus on reconstructing navigable spaces with real-world scan imperfections. Gibson introduced the concept of "Semantic Scene Understanding" and is often used for tasks like Object Goal Navigation in previously unseen spaces. It emphasizes the challenge of perception in cluttered, real-world settings.

iGibson is an extension of the Gibson Environment that adds full interactivity with hundreds of object models. It simulates physics-based interactions (e.g., pushing, grasping) and includes task-oriented benchmarks like BEHAVIOR. For researchers using Habitat for navigation, iGibson represents the next step towards full embodied task completion where navigation must be combined with object manipulation.

Room-to-Room (R2R) is the foundational Vision-and-Language Navigation (VLN) benchmark built on Matterport3D data within Habitat. It defines the core task: an agent must follow a natural language instruction to navigate from a start to a goal location. Key attributes:

• Uses human-written instructions.
• Evaluated primarily with Success weighted by Path Length (SPL).
• Serves as the standard testbed for instruction grounding and cross-modal alignment research.

REVERIE (Remote Embodied Visual Referring Expression in Real Indoor Environments) is a more advanced benchmark that requires an agent to navigate to and locate a target object specified by a concise, high-level language instruction (e.g., 'Bring me the blue mug on the desk'). It adds the challenge of fine-grained visual grounding to navigation, requiring the agent to identify a specific object instance from a category, often in a distant or obscured location.