Embodied Question Answering (EQA)

This module processes raw visual input from the agent's first-person perspective. It typically involves:

• Object Detection & Recognition: Identifying entities (e.g., 'refrigerator', 'apple') using models like Faster R-CNN or DETR.
• Semantic Segmentation: Labeling each pixel with a category (e.g., 'floor', 'wall', 'countertop') to understand navigable space and object boundaries.
• Depth Estimation: Inferring the 3D structure of the scene, often from RGB-D sensors or monocular depth prediction networks, crucial for navigation planning. Its output is a structured representation of the immediate scene, which feeds into the agent's internal world model.

This component translates high-level goals (e.g., 'go to the kitchen') into a sequence of low-level actions (e.g., move forward, turn left). Key elements include:

• Mapping: Building and updating an internal representation of the explored environment, often as a top-down 2D grid or a 3D voxel map.
• Localization: The agent's ability to track its own position within the built map.
• Planner: An algorithm (e.g., A*, Dijkstra's, or a learned policy) that calculates the optimal path from the current location to a target, avoiding obstacles. In EQA, the target is often unknown initially and must be inferred from the question.

This is the core reasoning bridge between language and perception. It performs two critical functions:

• Semantic Parsing: Decomposing the natural language question (e.g., 'What color is the mug on the table?') into an executable program or a set of constraints. This may identify the target object ('mug'), its location constraint ('on the table'), and the required attribute ('color').
• Visual Grounding: Linking the parsed linguistic concepts to specific visual entities in the environment. For 'mug on the table', the agent must identify all tables in its memory, navigate to them, and locate mugs in the scene. This relies heavily on visual relationship detection.

A dynamic memory system that stores information gathered during exploration. It is essential because the agent cannot see the entire environment at once. This includes:

• Spatial Memory: A record of visited locations, their layout, and objects found there.
• Episodic Memory: A log of past actions, observations, and their outcomes.
• Semantic Memory: Facts learned about the environment (e.g., 'the blue mug is in the kitchen'). This memory allows the agent to answer questions that require information from multiple locations without needing to re-navigate and enables efficient information gathering strategies.

The low-level controller that executes the discrete or continuous actions output by the navigation and policy modules. In simulated environments like AI2-THOR or Habitat, this interface translates abstract commands into API calls that the simulator understands. Typical action primitives include:

• Navigation Actions: MoveAhead, RotateLeft, RotateRight, LookUp, LookDown.
• Interaction Actions: Pickup, Open, Close, Slice for manipulating objects. The fidelity of this interface determines the agent's ability to interact with the world to gather necessary information (e.g., opening a fridge to see inside).

The final component that synthesizes the evidence into a natural language response. After the agent has executed its navigation and perception plan, this module:

• Aggregates Evidence: Combines visual observations from one or multiple viewpoints.
• Reasoning: Performs any required inference (e.g., counting objects, comparing attributes).
• Response Formulation: Generates a concise, textual answer (e.g., 'blue', 'two', 'yes'). While often a simple classifier or template for predefined question types, in advanced setups it can be an MLLM that generates free-form answers based on the agent's visual history and the original question.

Agents must reason about partially observable scenes where objects are hidden or visually ambiguous. Perceptual aliasing occurs when different locations or objects appear similar, confusing the agent's spatial memory. Occlusion reasoning is required to infer the presence of objects behind others or in drawers. For example, answering "Is there milk in the fridge?" requires the agent to approach the fridge and open it, understanding that the interior was initially occluded. This demands robust 3D scene understanding and the ability to interact with the environment to disambiguate.

Questions often require multi-step navigation and interaction sequences, creating a complex planning problem. The agent must decompose a high-level instruction like "What is on the table in the bedroom?" into a feasible action sequence: 1) Navigate to the bedroom, 2) Identify the table, 3) Approach it, 4) Visually scan its surface. This involves hierarchical planning under uncertainty, where failed actions (e.g., a blocked door) require re-planning. The credit assignment problem—determining which actions in a long sequence were critical for success—makes learning these policies difficult.

Natural language questions are often underspecified or context-dependent. The agent must resolve referential ambiguity (e.g., "the blue mug" when there are two) and interpret spatial relations (e.g., "next to," "behind"). This requires tight integration between the language understanding module and the visual grounding system. The challenge is to map linguistic concepts to actionable spatial goals in a continuous, dynamic environment, a process more complex than static Visual Question Answering (VQA). For instance, "Bring me the book you see near the sofa" requires identifying the sofa, searching its vicinity, and recognizing a book.

Most EQA research uses simulated environments like AI2-THOR or Habitat. Models trained in simulation often fail in real-world deployment due to the reality gap—differences in visual appearance, physics, and actuator control. Textures, lighting, and object dynamics are idealized in sim. Bridging this gap requires techniques like domain randomization (varying simulation parameters during training) or Sim2Real transfer learning. This is critical for practical applications, as collecting large-scale real-world EQA data with ground-truth actions and answers is prohibitively expensive and slow.

The agent must maintain and update a unified internal world model that fuses information across modalities:

• Visual features from first-person RGB-D frames.
• Spatial memory (e.g., a topological map or egocentric occupancy grid).
• Linguistic context from the question and dialog history. Aligning these heterogeneous representations into a coherent state for decision-making is a core architectural challenge. Poor integration leads to the agent "forgetting" the question after navigating or failing to link what it sees to the original query. This is a key focus of Vision-Language-Action (VLA) model design.

Learning effective navigation and interaction policies from scratch requires massive amounts of trial-and-error experience, making reinforcement learning (RL) approaches sample-inefficient. Furthermore, agents must generalize to novel environments, object arrangements, and phrasing of questions not seen during training. Achieving compositional generalization—understanding new combinations of known concepts (e.g., "red stove" when trained on "red pot" and "stove")—is particularly difficult. Techniques like imitation learning from expert demonstrations, pre-training on vision-language tasks, and modular network design are employed to improve data efficiency and robustness.

Visual Question Answering is a foundational multimodal task where a model must answer a natural language question based solely on the static content of a single input image. Unlike EQA, the agent is passive; it does not navigate or interact. The model performs visual recognition, scene understanding, and often commonsense reasoning to produce an answer.

• Key Difference from EQA: Static vs. Embodied. VQA answers from a given image; EQA requires active exploration to gather the necessary visual information.
• Architectural Basis: Many EQA systems use VQA models as a core component to answer questions once the relevant viewpoint is reached.
• Example: Given an image of a kitchen, answering "Is the refrigerator door open?"

Language-Guided Navigation is the core mobility sub-task within EQA. The agent receives a natural language instruction (e.g., "Go to the kitchen and find the red mug on the counter") and must navigate through a photorealistic simulated environment to reach a target location or object.

• Embodiment: The agent executes low-level actions like move_forward, turn_left, turn_right, and stop.
• Challenges: Requires understanding spatial language ("left of", "behind"), long-horizon planning, and dealing with partial observability.
• Benchmarks: Tasks like Vision-and-Language Navigation (VLN) in Matterport3D simulators focus purely on this navigation aspect.

Visual Grounding is the fundamental computer vision task of establishing a link between linguistic concepts (words or phrases) and specific spatial regions within a visual scene. It answers "where" a described object is located.

• Core to EQA: An EQA agent must ground the question's entities (e.g., "the blue book") in its visual perception to know what to look for.
• Related Tasks: Referring Expression Comprehension (REC) is a specific instantiation where a free-form description is used to localize an object.
• Output: Typically a bounding box or segmentation mask pinpointing the referred object.

Embodied AI is the overarching research field concerned with developing intelligent agents that learn and act within physical or simulated environments. It emphasizes that intelligence is shaped by interaction with a surrounding world.

• Core Principle: Perception → Cognition → Action loop.
• EQA's Role: EQA is a canonical embodied AI benchmark that integrates navigation, visual perception, and question answering.
• Broader Scope: Includes tasks like object manipulation, instruction following, and embodied navigation beyond just Q&A.
• Platforms: Heavily relies on simulators like AI2-THOR, Habitat, and Gibson for training and evaluation.

Sim-to-Real Transfer is the critical methodology of training embodied agents like EQA systems in high-fidelity simulated environments before deploying their learned policies to physical robots. It addresses the cost, safety, and scalability limitations of real-world training.

• Domain Gap: The difference between simulation and reality (e.g., lighting, textures, physics) that can degrade performance.
• Techniques: Used to bridge the gap include domain randomization (varying sim parameters) and domain adaptation.
• Relevance to EQA: All major EQA benchmarks (e.g., on Matterport3D scans) are conducted in simulation, with the long-term goal of transferring capabilities to real robots.

A Multimodal Large Language Model is a foundation model that extends the reasoning and generative capabilities of an LLM to process and understand multiple input modalities, such as images and text. MLLMs are becoming the central architecture for advanced EQA systems.

• Function in EQA: The MLLM acts as the agent's "brain," processing visual observations from the environment, interpreting the question, maintaining memory, and deciding on actions.
• Architecture: Typically uses a visual encoder (like ViT) to convert images into embeddings, which are projected into the LLM's token space.
• Capability: Enables in-context learning and chain-of-thought reasoning for complex, multi-step EQA tasks.