RT-1/RT-2 Architecture

The core architectural innovation is the tokenization of all input and output modalities into a single, unified sequence. This includes:

• Visual tokens: Raw camera images are processed by a pre-trained Vision Transformer (ViT) to produce a sequence of patch tokens.
• Language tokens: Natural language instructions are tokenized using a standard text tokenizer.
• Action tokens: Robot actions (e.g., end-effector poses, gripper commands) are discretized and represented as a sequence of discrete tokens. This unified representation allows a standard transformer decoder to process the multimodal context and autoregressively predict the next action token, treating robot control as a sequence modeling problem.

Both architectures employ a causal transformer decoder as the central neural network backbone. This model:

• Takes the concatenated sequence of image tokens and language tokens as the conditioning context.
• Autoregressively predicts the sequence of action tokens, one token at a time.
• Leverages the self-attention mechanism to build rich representations that fuse visual and linguistic information. This design enables the model to learn from large, diverse datasets of robot experiences, generalizing across tasks, environments, and object instances seen during training.

RT-2 introduced a significant advancement by co-fine-tuning a pre-trained Vision-Language Model (VLM). Instead of training from scratch, RT-2 initializes its weights from a massive VLM like PaLM-E or PaLI-X, which has been pre-trained on vast datasets of web images and text. This process, known as vision-language-action (VLA) transfer, imbues the model with semantic knowledge and visual reasoning capabilities from the internet. The model is then fine-tuned on robot data, allowing it to interpret commands like 'pick up the extinct animal' and correctly identify a dinosaur toy, demonstrating emergent semantic reasoning.

To integrate continuous robot actions into the token-based sequence model, a discretization strategy is critical.

• RT-1 used binning, where each dimension of the action vector (e.g., x, y, z, rotation) is quantized into a fixed number of bins, each assigned a token ID.
• RT-2 can use more sophisticated methods like VQ-VAE (Vector Quantized Variational Autoencoder) to learn a discrete codebook for compressing continuous action trajectories. This creates a vocabulary of action tokens. During inference, the model's predicted token IDs are mapped back to continuous values via the codebook or bin centers to command the robot.

The architecture's generalist capability is enabled by training on massive, heterogeneous datasets of robot demonstrations. For RT-1, this was the 130k-episode Google Robots dataset collected from 13 different robot embodiments across 700+ tasks. The training objective is simple behavior cloning (supervised learning) on this data. The transformer learns to associate the diverse visual scenes and language instructions with the corresponding action sequences, absorbing patterns of effective manipulation and navigation. This scale is essential for the model to exhibit robust multi-task performance and positive transfer between skills.

A key capability unlocked by the VLM backbone in RT-2 is in-context reasoning. The model can perform internal chain-of-thought before outputting actions. For example, when asked to 'move the coke can to the top of the stack,' the model can reason internally about object relationships and stability. This is facilitated by the pre-training on web data, which includes logical and sequential reasoning tasks. The architecture does not explicitly output text reasoning steps, but the internal representations within the transformer layers enable this multi-step semantic planning, bridging high-level intent to low-level control.

The core capability of RT-1 is large-scale multi-task learning. Trained on a dataset of 130k real-world robotic trials, a single RT-1 model can execute over 700 distinct tasks. This includes:

• Object pick-and-place across diverse geometries.
• Opening drawers and doors with varied mechanisms.
• Placing objects into specific receptacles (e.g., a cup in a rack).
• Wiping surfaces and pressing buttons. The model's token-based action representation allows it to condition its output on a natural language instruction and the current image, switching seamlessly between tasks without task-specific modules.

RT-2's key advancement is leveraging knowledge from web-scale vision-language data for symbolic understanding and reasoning. This enables capabilities unseen in RT-1:

• Interpreting abstract concepts: It can execute 'place the bag in the correct recycling bin' by identifying the 'recycling' symbol.
• Mathematical reasoning in context: It can perform 'pick up the object with the number 3 on it' from a cluttered scene.
• Metaphorical understanding: Commands like 'move the apple to the winner' (where 'winner' is a trophy) are parsed by grounding language in visual semantics learned from the internet, not just robotic data.

Both architectures demonstrate the ability to decompose and execute chained instructions. For example, given 'bring me a Coke from the fridge,' the model must:

1. Navigate to and identify the refrigerator.
2. Open the fridge door.
3. Locate and grasp the Coke can.
4. Close the fridge door.
5. Bring the can to the human. This is achieved through the model's temporal context window, which processes a short history of images and actions, allowing it to maintain the state of a multi-step plan without an explicit symbolic planner.

A critical real-world test is performance amidst visual clutter and object appearance variation. RT models are evaluated on their ability to ignore irrelevant objects and execute tasks under conditions like:

• Novel backgrounds and lighting not seen in training.
• Distractor objects placed near the target.
• Partial occlusions of the target object. The FiLM conditioning mechanism in RT-1 and the co-attention in RT-2's vision-language backbone allow the model to focus its visual processing on instruction-relevant elements, demonstrating significant robustness compared to prior methods.

While trained on a fixed set of objects, the architectures show emergent adaptation capabilities. By providing a single demonstration (via an image sequence of the new task) or a few language-described examples at inference time, the model can often generalize to:

• Manipulating novel object categories with similar affordances.
• Performing new combinations of known skills (e.g., 'sweep the spilled beans into the dustpan'). This is a direct benefit of the sequence-to-sequence formulation and the rich semantic features from pre-trained components, treating new scenarios as a form of in-context learning.

The token-based design facilitates transfer across domains. Demonstrated and ongoing research includes:

• Sim-to-Real Transfer: Policies trained in photorealistic simulators (using simulated images tokenized the same way) can transfer to physical robots with minimal fine-tuning, as the input representation is consistent.
• Cross-Embodiment Learning: By normalizing actions to a shared token space, there is potential to train on data from multiple robot platforms (e.g., different arm types) to create a more universal policy, a direction explored in successors like RT-X.

A Vision-Language-Action (VLA) model is a multimodal AI architecture that directly processes visual inputs and natural language instructions to generate low-level physical actions or control commands for a robot. It represents the core architectural paradigm of RT-1 and RT-2, where a single transformer model unifies perception, language understanding, and action generation into one end-to-end system. This contrasts with modular pipelines that separate these components.

• Key Feature: Unifies three modalities (vision, language, action) in a single token stream.
• Example: RT-2 directly outputs robot arm joint angles or end-effector velocities based on a camera image and the command "pick up the blue block."

An embodied foundation model is a large-scale, pre-trained neural network designed to serve as a general-purpose backbone for a wide range of robotic tasks by integrating perception, reasoning, and action generation. RT-2 is a prime example, built by fine-tuning a massive vision-language model (PaLI-X) on robot data. This approach leverages internet-scale pre-training for visual and linguistic concepts, then grounds this knowledge in physical action.

• Core Idea: Transfer knowledge from web-scale vision-language pre-training to the robotics domain.
• Benefit: Exhibits emergent capabilities like reasoning about object categories not seen in robot training data.

Multimodal instruction tuning is the fine-tuning process where a pre-trained vision-language model is trained on datasets of (image, instruction, action) triplets to align its outputs with executable robot behaviors. This is the critical adaptation step used to create RT-2 from PaLI-X. The model learns to interpret instructions in the context of the current visual scene and output a sequence of action tokens.

• Process: Takes a frozen VLM backbone and trains it on robot trajectories.
• Output Space: Actions are discretized into a vocabulary of tokens (e.g., "binarized" continuous values), allowing the language model's text decoder to generate them.

End-to-end visuomotor control is an approach where a single neural network model learns to directly map raw visual observations (pixels) to low-level robot motor commands, without explicit intermediate representations like state estimation or planning. RT-1 and RT-2 exemplify this paradigm. The model implicitly learns necessary representations (like object pose or affordances) within its layers, optimizing the entire pipeline for the final control objective.

• Advantage: Reduces engineering complexity of hand-designed state estimators.
• Challenge: Requires large, diverse datasets to learn robust representations from pixels alone.

PaLM-E is an embodied multimodal model from Google that integrates a large language model (PaLM) with a vision transformer, using a single token stream to process language, visual inputs, and sensor data for planning and control tasks. It is a direct architectural predecessor to RT-2. PaLM-E demonstrated that injecting continuous sensor observations (like images) into the token stream of a large language model could enable embodied reasoning and planning, paving the way for RT-2's more direct action generation.

• Key Differentiator: Focused more on high-level planning and reasoning ("what to do") whereas RT-2 focuses on low-level control ("how to do it").
• Input: Language commands and multi-image sequences.

The SayCan paradigm is a robotics framework from Google that combines a large language model's high-level task planning capability ('Say') with a learned affordance model's understanding of feasible low-level skills ('Can') to execute instructions in the real world. It represents a complementary, hierarchical approach to RT-1/RT-2's end-to-end method. SayCan uses LLMs for decomposition and skill selection, while RT-2 generates actions directly.

• Architecture: Modular. LLM proposes skills, value function checks feasibility.
• Comparison to RT-2: SayCan is more interpretable and modular; RT-2 is more general and requires less predefined skill engineering.