VoxPoser System

VoxPoser uses a large language model (LLM) as a spatial reasoner to interpret natural language instructions and decompose them into a sequence of spatial constraints and affordance functions. The LLM does not process pixels directly; instead, it outputs code-like programs that define relationships in 3D space.

• Constraint Generation: For an instruction like 'put the apple in the bowl,' the LLM generates constraints such as Graspable(apple) and Inside(apple, bowl).
• Function Composition: These textual constraints are translated into executable functions that query a pre-trained vision-language model (VLM) to score voxel locations in the scene.

A pre-trained vision-language model (VLM), such as CLIP or OWL-ViT, acts as a visual scorer. It grounds the LLM's abstract constraints into the robot's observed 3D voxel grid.

• Query Execution: For a constraint like Graspable(apple), the system renders a 2D view from the perspective of each candidate voxel and queries the VLM with the text prompt 'a graspable handle of an apple'.
• Probability Map: The VLM's output scores are aggregated into a 3D value map, where each voxel holds a value representing the likelihood it satisfies the constraint (e.g., being a good grasping point).

The system's core innovation is the composition of multiple 3D maps via logical and temporal operators to form a complete task plan.

• Value Maps (V-Maps): Represent where an object or relationship should be (e.g., the target location for placement).
• Affordance Maps (A-Maps): Represent how the robot can interact with a location (e.g., a gripper orientation for pushing or grasping).
• Temporal Logic: Operators like AND, THEN, and WHILE combine maps. For 'push the coke can to the coaster,' it composes Graspable(coke_can) THEN On(coke_can, coaster).

The composed 3D maps create an optimization landscape for a motion planner. The planner finds a robot trajectory that maximizes the combined value across all constraints.

• Trajectory Optimization: The planner treats the voxel maps as a cost function. It searches for end-effector waypoints that pass through high-value regions in the affordance map (for interaction) and terminate in high-value regions in the value map (for the goal state).
• No Task-Specific Training: This optimization happens at inference time. No reinforcement learning or demonstration data for the specific task is required, enabling zero-shot generalization to novel instructions.

VoxPoser generates high-level waypoints in task space (Cartesian coordinates). These are fed into a standard robotic control stack for safe, smooth execution.

• Output: A sequence of end-effector poses (position + orientation).
• Downstream Execution: A motion planner (e.g., OMPL, MoveIt) checks for collisions and finds a feasible joint trajectory. A low-level controller (e.g., PID, impedance control) then executes the trajectory on the physical robot.
• Modularity: This separation allows VoxPoser to be robot-agnostic, interfacing with different arms and grippers through their native APIs.

Because it relies on pre-trained foundation models (LLMs/VLMs) and requires no task-specific fine-tuning, VoxPoser demonstrates strong sim-to-real transfer capabilities.

• Foundation Model Prior: The LLM and VLM provide a rich, pre-existing understanding of objects, geometry, and language that transfers directly from internet-scale training to the physical world.
• Reduced Reality Gap: The system does not learn low-level control dynamics that are simulation-specific. Its output is spatial waypoints, which are easier for a real robot's controller to track accurately compared to raw torque commands.
• Generalization: It can interpret instructions for objects and scenes it was never explicitly trained on in a robotics context, leveraging the broad knowledge of the foundation models.

A Vision-Language-Action (VLA) model is a multimodal AI architecture that directly processes visual inputs and natural language instructions to generate low-level physical actions or control commands for a robot. Unlike systems like VoxPoser that compose intermediate 3D maps, VLAs often learn an end-to-end mapping from pixels and text to motor torques or joint velocities. They are typically trained on large datasets of robot interaction data.

• Key Function: Closes the loop from perception to actuation in a single model.
• Example: RT-2, a transformer-based model that tokenizes images, instructions, and actions.

Affordance prediction is the task of identifying, from visual input, the potential ways an object or scene region can be interacted with. VoxPoser generates affordance maps in voxel space, highlighting where actions like 'grasp', 'push', or 'place' are feasible. This is distinct from pure object detection, as it infers action possibilities.

• Core Concept: Grounds abstract actions in the physical geometry of the environment.
• VoxPoser's Role: Uses a Vision-Language Model (VLM) to score voxels based on language queries like 'where can it be grasped?'

3D visual grounding is the task of localizing a natural language query within a 3D representation of a scene, such as a point cloud or voxel grid. This is a foundational capability for VoxPoser, which must ground instructions like 'move the mug to the table' into specific 3D coordinates. It requires understanding both spatial relationships and object semantics.

• Input: A 3D scene representation and a text query.
• Output: A set of 3D coordinates or a segmentation mask for the referred entity.

Hierarchical task planning is a method where a high-level planner decomposes a complex language instruction into a sequence of executable sub-tasks or skills. VoxPoser employs this via a Large Language Model (LLM), which breaks down 'make coffee' into steps like 'find kettle', 'grasp kettle', 'pour water'. The low-level execution is then handled by the composed value/affordance maps.

• Abstraction Layer: Separates symbolic reasoning from geometric motion planning.
• Synergy: Combines the world knowledge of LLMs with the physical understanding of VLMs.

In the context of VoxPoser, a value map is a 3D grid (voxel space) where each voxel is assigned a scalar value representing its utility for achieving a task sub-goal. For the instruction 'push the block to the red area', a value map would assign high values to voxels in the red area. It provides a gradient that a motion planner can follow.

• Generation: Created by querying a VLM with language prompts about task success.
• Purpose: Guides the robot by defining 'what is good' spatially, enabling optimization of trajectories toward high-value regions.

The SayCan paradigm is a robotics framework that combines a large language model's high-level task planning capability ('Say') with a learned affordance model's understanding of feasible low-level skills ('Can'). VoxPoser is a direct intellectual descendant, replacing the pre-trained skill affordance model with zero-shot compositions from VLMs and LLMs.

• Key Innovation: Grounds LLM plans in physically feasible actions.
• Comparison: SayCan uses pre-trained skills; VoxPoser generates skills on-the-fly via 3D map composition.