Embodied Question Answering (EQA)

The Perception Module is responsible for processing raw sensor data, typically first-person (egocentric) visual streams, into a structured representation of the agent's immediate surroundings. This involves:

• Semantic Segmentation: Labeling each pixel in the image with an object category (e.g., 'chair', 'door').
• Depth Estimation: Inferring the distance from the agent to surfaces in the scene.
• Object Detection & Recognition: Identifying and localizing discrete entities.
• Affordance Prediction: Identifying potential interaction points on objects (e.g., a handle is 'graspable'). This processed scene representation is the primary input for the agent's navigation and reasoning systems.

The Navigation Policy is a control function that decides the agent's low-level movement actions (e.g., 'move forward 0.25m', 'turn left 30 degrees') based on the current perceptual input and a high-level goal. For EQA, this goal is derived from the question. Key approaches include:

• Classical Path Planning: Using the scene representation to build a map and plan a collision-free path.
• Learning-Based Policies: Training a neural network (often via Reinforcement Learning or Imitation Learning) to map visual observations directly to actions. This policy must handle partial observability, as the agent only sees a fraction of the environment at any time.

This component interprets the natural language question and determines the epistemic actions required to answer it. It performs spatial and relational reasoning to formulate a plan. For example, the question "What color is the mug on the desk in the bedroom?" requires:

1. Spatial Decomposition: Identify target room ('bedroom'), then target furniture ('desk'), then target object ('mug').
2. Action Sequence Planning: Generate sub-goals: 'Navigate to bedroom', 'Locate desk', 'Inspect mug'.
3. Answer Type Inference: Determine the answer is a color attribute. Modern systems often use a Large Language Model (LLM) or a Vision-Language Model (VLM) for this high-level symbolic planning.

Because the agent's view is limited, it must maintain an internal Memory & State Estimator. This component builds and updates a persistent world model as the agent explores. It integrates:

• Episodic Memory: A record of previously visited locations and seen objects (e.g., 'I already checked the kitchen counter').
• Spatial Memory: A mental map of the environment's layout, often built using techniques from Simultaneous Localization and Mapping (SLAM).
• Task State Tracking: Keeping track of progress towards the question's sub-goals (e.g., 'desk found, now searching for mug'). This prevents redundant exploration and enables efficient information gathering.

Once the agent has executed the necessary navigation and interaction to gather visual evidence, the Answer Generation Module synthesizes the final response. This is not merely retrieving a fact from a database; it involves grounding the answer in the agent's direct sensory experience. The process includes:

• Evidence Aggregation: Combining information from multiple viewpoints or interactions (e.g., seeing a mug from the front and side to confirm its color).
• Visual Question Answering (VQA): Applying a VQA model to the specific visual observation containing the answer.
• Response Formulation: Producing a concise, natural language answer (e.g., "The mug is blue.") that is verifiably grounded in the agent's perceptual history.

The seminal Embodied Question Answering (EQA) v1 dataset, introduced by Das et al. in 2018, established the core task. It is built on the House3D simulation environment, which contains over 45,000 synthetic indoor scenes. The dataset features:

• Question Types: Location ('Where is the toilet?'), Color ('What color is the couch?'), and Place ('Is the bedroom upstairs?')
• Agent State: The agent begins at a random spawn point and must answer egocentric, text-based questions about objects in other rooms.
• Primary Metric: Navigation Success (reaching the target room) and Answering Accuracy (providing the correct answer). This dataset highlighted the challenge of temporal reasoning, as agents must remember and integrate information gathered over a trajectory.

An evolution of the original task, EQA-3D (or EQAv2) introduced more complex, compositional questions and a focus on object-level grounding. Key characteristics include:

• Question Compositionality: Queries like 'What is the color of the lamp on the desk in the study?' require multi-step reasoning about object relationships.
• Fine-Grained Answers: Answers are specific object attributes (color, shape) or counts, not just room names.
• Benchmark Focus: It measures an agent's ability to perform visual grounding—linking linguistic phrases to specific objects—after navigation. This shifted the field's emphasis from simple room-finding to detailed scene understanding and memory.

To increase realism, the Matterport3D EQA (MP3D-EQA) benchmark uses photorealistic 3D scans of real buildings from the Matterport3D dataset. This introduced the 'Sim-to-Real' gap into EQA evaluation.

• Environmental Fidelity: Contains 90 building-scale environments with realistic textures, lighting, and clutter.
• Challenges: Agents must handle visual noise, occlusion, and the long-tailed distribution of objects found in real homes.
• Impact: Performance on MP3D-EQA is often lower than on synthetic datasets like House3D, providing a more rigorous test of an agent's generalization capabilities and the robustness of its visual representations.

Interactive Question Answering (IQA) extends EQA by requiring the agent to manipulate the environment to gather information. This addresses a key limitation of classic EQA where answers are based on passive observation.

• Core Mechanic: Questions like 'Is the stove on?' or 'What is written on the paper under the book?' cannot be answered without physical interaction (e.g., turning a knob, moving an object).
• Action Space Expansion: The agent's action set includes navigation and manipulation primitives (e.g., Open, Pickup, Toggle).
• Benchmark Significance: IQA benchmarks, such as those in AI2-THOR, evaluate an agent's ability to sequence perception-action loops and understand object affordances, moving closer to true embodied intelligence.

EQA performance is quantified through a suite of complementary metrics, each diagnosing different agent capabilities:

• Navigation Success (NS): Binary measure of whether the agent reached the target location. Isolates path planning competence.
• Answer Accuracy (AnsAcc): Percentage of questions answered correctly, regardless of navigation success. Measures visual question answering skill.
• Oracle Answer Accuracy (Oracle): Accuracy if the agent is teleported to the optimal viewpoint. Reveals the upper bound of the VQA component.
• Success weighted by Path Length (SPL): Combines navigation success with efficiency: SPL = Success * (Optimal Path Length / Actual Path Length). Penalizes meandering.
• Distance to Goal (DTG): The average shortest-path distance remaining when the agent stops. Useful for analyzing partial success in failed episodes.

A Vision-Language-Action (VLA) model is the core architecture that enables EQA. It is a multimodal AI system that directly processes visual inputs (e.g., camera feeds) and natural language instructions (the question) to generate low-level physical actions or control commands for a robot. Unlike models that only answer questions about static images, VLAs are trained to output motor commands like 'move forward 0.5m' or 'turn left 90 degrees' to gather the necessary information.

• Key Function: Translates 'what is' into 'how to find out.'
• Example: A VLA model powers the agent that hears 'What color is the mug on the coffee table?' and outputs navigation commands to find the table, then a manipulation command to look at the mug.

Visual Language Navigation (VLN) is a foundational subtask for many EQA scenarios. It involves directing a robotic agent to follow natural language navigation instructions within a photorealistic environment, using only egocentric visual input. The agent must interpret commands like 'Go down the hall and stop before the second door on the left' and execute the corresponding path.

• Core Challenge: Grounding spatial language (e.g., 'left,' 'second,' 'before') in a continuously changing visual stream.
• Relation to EQA: VLN is often the first phase of an EQA task, where the agent must navigate to the relevant location before answering a question about it.

Egocentric perception refers to computer vision and sensing from a first-person perspective, as mounted on a robot or wearable device. This is the primary data modality for embodied agents. It contrasts with third-person or static scene understanding.

• Key Characteristics: The field of view is limited and dynamic, objects may be occluded or only partially visible, and the scene changes with every movement.
• Importance for EQA: Questions are answered based on what the agent can see from its current viewpoint, not an omniscient map. The agent must actively control this viewpoint to gather complete information.

Hierarchical task planning is a reasoning method where a high-level planner decomposes a complex instruction (like an EQA question) into a sequence of executable sub-tasks or skills. For the question 'Is there milk in the refrigerator?', a planner might generate: 1. Navigate to the kitchen, 2. Locate the refrigerator, 3. Open the door, 4. Scan shelves, 5. Answer yes/no.

• Role in EQA: Provides the 'plan' that bridges the question to a series of low-level actions. It is often implemented using large language models (LLMs) for commonsense reasoning.
• Frameworks: The SayCan paradigm is a prominent example, where an LLM ('Say') proposes skills that are filtered by an affordance model ('Can') to ensure physical feasibility.

Sim-to-Real transfer is the critical engineering process of training and validating EQA agents in high-fidelity physics-based simulations (like NVIDIA Isaac Sim) before deploying them on physical hardware. This is essential because collecting vast 'embodied datasets' in the real world is prohibitively expensive and slow.

• The Reality Gap: The challenge that simulation physics, visuals, and sensor noise never perfectly match reality.
• Techniques: Involves domain randomization (varying textures, lighting, physics parameters in sim) and domain adaptation to make models robust to discrepancies, ensuring an agent trained in a virtual apartment can operate in a real one.

An embodied foundation model is a large-scale, pre-trained neural network designed as a general-purpose backbone for a wide spectrum of robotic tasks, including EQA. Models like RT-2 and PaLM-E are trained on massive datasets of internet-scale language and vision data, combined with robot interaction data.

• Capability: Exhibits emergent abilities like spatial reasoning, affordance understanding, and instruction following that can be specialized for EQA with minimal additional training.
• Architecture: Typically a transformer that tokenizes images, language, and actions into a single stream, allowing it to process multi-modal history and predict future actions autoregressively.