AI2-THOR

The framework supports the simulation of multiple embodied agents within the same scene, enabling research into collaborative and adversarial tasks. This is crucial for studying multi-agent coordination and communication.

It also facilitates human-in-the-loop data collection and evaluation:

• Expert Demonstrations: Humans can control an avatar to generate optimal trajectories for imitation learning.
• Interactive Evaluation: Researchers can manually guide or evaluate agents in real-time.
• Crowdsourcing: The visual fidelity and intuitive interaction model allow for scalable data collection via platforms like Amazon Mechanical Turk.

Beyond raw pixels, AI2-THOR provides a comprehensive structured metadata layer. Each scene is represented as a graph of objects with rich properties, enabling symbolic reasoning and planning.

Metadata includes:

• Semantic Properties: Object category (e.g., Mug), material, size.
• Spatial Properties: 3D bounding box, position, rotation.
• Relational Properties: Receptacle relationships (e.g., a Mug is on a Table), object states.
• Affordances: What actions are currently valid for each object.

This allows for hybrid agents that combine neural perception with classical symbolic planning over this object graph.

For rigorous evaluation of generalization, AI2-THOR supports extensive scene and task customization.

• Scene Variability: Object textures, layouts, and lighting conditions can be modified.
• ProcTHOR: A companion framework for procedural generation of massive, diverse training scenes to combat overfitting.
• Task Definition: Researchers can script complex task sequences with custom reward functions and goal conditions.
• Asset Integration: New 3D object models can be imported, provided they are annotated with the required metadata and affordances.

This flexibility is key for studying sim-to-real transfer and zero-shot generalization to novel environments.

Success weighted by Path Length (SPL) is the primary quantitative metric for evaluating navigation agents. It refines a simple success rate by penalizing inefficiency.

• Formula: SPL = (1 / N) * Σ (S_i * (L_i / max(P_i, L_i)))
• S_i: Success (1 or 0) for trial i.
• L_i: Length of the optimal (shortest) path.
• P_i: Length of the agent's path. An agent that succeeds but takes a circuitous route receives a score less than 1.0. SPL is the standard reporting metric for benchmarks like R2R run in AI2-THOR and Habitat, providing a holistic measure of both effectiveness and efficiency.

Sim-to-Real Transfer is the fundamental challenge of deploying policies trained in simulation (like AI2-THOR) onto physical robots in the real world. The reality gap—differences in visuals, physics, and sensor noise—can cause severe performance drops. Techniques to bridge this gap include:

• Domain Randomization: Varying textures, lighting, and object properties in simulation to improve robustness.
• System Identification: Tuning simulation parameters to match real-world dynamics.
• Learning Domain-Invariant Features: Training models to focus on semantic cues rather than superficial visual details. Success in sim-to-real is critical for the practical application of AI2-THOR-trained agents.

A Language-Conditioned Policy is the core neural network controller for a language-guided agent. It is a function π(a_t | o_t, g) that outputs an action a_t (e.g., move_ahead, pickup) given the current visual observation o_t and a goal g encoded from a natural language instruction. Architectures for these policies often involve:

• A visual encoder (e.g., CNN) for processing egocentric RGB-D frames.
• A language encoder (e.g., BERT, LSTM) to embed the instruction.
• A fusion module (e.g., Cross-Modal Transformer) to combine modalities.
• An action decoder (MLP or GRU) to produce the final command. This policy is the primary model trained within AI2-THOR environments.