SayCan Paradigm

The 'Say' component is a large language model (LLM) responsible for high-level task decomposition and planning. Given a natural language instruction (e.g., 'I spilled my drink, can you help?'), the LLM generates a sequence of plausible sub-tasks. It operates purely in the linguistic domain, leveraging its world knowledge and common-sense reasoning to propose steps like '1. Find a sponge, 2. Pick up the sponge, 3. Wipe the spill.' Its output is a probability distribution over possible next steps, representing what the robot should do in an ideal world.

The 'Can' component is a learned value function or affordance model that evaluates what the robot is able to do given the current physical context. It takes the robot's sensory state (e.g., an image from its camera) and a proposed skill (e.g., 'pick up sponge') and outputs a probability score. This score represents the feasibility of successfully executing that skill from the current state. It grounds the LLM's abstract plans in physical reality, preventing the robot from attempting impossible actions like picking up an object that is not visible or within reach.

The core innovation of SayCan is the multiplicative fusion of the outputs from the 'Say' and 'Can' modules. For each candidate skill proposed by the LLM, the system calculates a combined score: P(skill) = P_Say(skill | instruction, history) * P_Can(skill | state). The skill with the highest product is selected for execution. This ensures the chosen action is both semantically relevant to the instruction and physically feasible. The process repeats after each action, creating a closed-loop plan-and-act cycle.

SayCan relies on a predefined library of low-level robotic skills (e.g., Pick(), Place(), NavigateTo(), Open()). Each skill is associated with a trained policy—often a neural network—that executes the skill by mapping raw sensor observations to low-level motor commands. The 'Can' module's score for a skill is typically derived from the success probability predicted by that skill's value function. This modular design allows the system to leverage a mix of skills trained via reinforcement learning, imitation learning, or classical methods.

In deployment, SayCan operates in a real-time, closed-loop manner:

• Perceive: The robot captures the current scene (e.g., an image).
• Plan & Score: The LLM ('Say') proposes skills, and the affordance model ('Can') scores them based on the image.
• Select & Act: The highest-scoring skill is sent to its corresponding low-level policy for execution.
• Observe & Repeat: The robot observes the new state, and the cycle repeats until the high-level instruction is complete. This enables long-horizon tasks through iterative re-planning, adapting to changes in the environment.

SayCan's primary contribution is a principled framework for grounding LLMs in physical action. Key innovations include:

• De-risking LLM Plans: By filtering LLM outputs through a feasibility model, it significantly reduces the execution of nonsensical or dangerous actions.
• Separation of Concerns: Decouples linguistic reasoning (which benefits from internet-scale data) from physical grounding (which requires robot interaction data).
• Generalization: The LLM's knowledge allows the system to interpret and plan for novel instructions not seen in robot training data.
• Foundational Paradigm: It established a blueprint for later Vision-Language-Action (VLA) models like RT-2 and PaLM-E, which integrate these components into a single end-to-end model.

The LLM ('Say') decomposes the instruction into logical sub-tasks: 1. Find a snack. 2. Find a drink. 3. Bring both to the user. The affordance model ('Can') evaluates each step against the robot's current visual scene and its learned skills. It might output high probabilities for pick_up(chips_bag) and pick_up(water_bottle) but a near-zero probability for pick_up(apple) if no apple is visible. SayCan selects the sequence with the highest combined P(skill feasible) * P(skill relevant), ensuring the robot only attempts physically possible actions.

Faced with an ambiguous instruction, the LLM proposes plausible skill sequences like [find(trash), pick_up(trash), place_in(bin), find(dishes), pick_up(dishes), place_in(sink)]. The affordance model grounds this plan by evaluating each skill against the robot's live camera feed. It identifies specific objects (e.g., a paper_cup on the table) and assigns a high feasibility score to pick_up(paper_cup). If the trash bin is occluded, place_in(bin) receives a low score, causing the LLM to replan or the robot to first navigate for a better view. This tight loop enables robust handling of open-ended tasks.

This demonstrates hierarchical planning under physical constraints. The LLM might generate a long chain: [find(mug), pick_up(mug), find(coffee_machine), navigate_to(coffee_machine), place_mug_under(spout), press_button(brew)]. The affordance model acts as a per-step feasibility filter. If the coffee machine's 'brew' button is not visible or the machine is off, the probability for press_button(brew) plummets. SayCan can then either halt for human assistance or the LLM can insert a prerequisite skill like plug_in(coffee_machine). This prevents the robot from blindly executing a logically sound but physically impossible plan.

This highlights the system's real-time grounding. The initial plan may be [find(watering_can), pick_up(watering_can), navigate_to(plant), pour(plant)]. If the robot picks up the can and finds it empty, the affordance model's score for pour(plant) becomes zero when conditioned on the new visual state. This low 'Can' score signals the LLM to replan. The LLM might then insert navigate_to(sink) and fill(watering_can) before re-attempting the pour. This dynamic interplay allows the system to recover from unexpected world states without explicit pre-programming for every contingency.

The 'Can' component is fundamentally a learned safety and feasibility prior. It is trained on robot interaction data to estimate the probability a low-level skill (e.g., pull(drawer)) will succeed given the current visual observation. This prevents dangerous or futile actions. For example:

• Obstacle Avoidance: A high score for navigate_to(table) only if the path is clear.
• Grasp Validation: A high score for pick_up(glass) only if a stable grip is geometrically plausible.
• State Checking: A near-zero score for open(fridge) if the robot's gripper is already full. This model provides the physical common sense that pure LLMs lack.

SayCan is a planning framework, not a monolithic model. It is designed to integrate with other core components of embodied intelligence:

• VLA Models as Skill Controllers: A Vision-Language-Action model like RT-2 can serve as the low-level policy that executes the skills (e.g., pick_up) selected by SayCan.
• Hierarchical Task Planning: SayCan's LLM performs the high-level decomposition, which can be refined by a dedicated task planner.
• Sim2Real Transfer: The affordance model ('Can') can be trained in simulation (e.g., using a physics engine) and transferred to a real robot, helping bridge the reality gap.
• Imitation Learning: The skill library used by 'Can' is often populated via behavior cloning on datasets of human demonstrations.

Affordance prediction is the core 'Can' component of the SayCan paradigm. It is the perceptual task of identifying, from visual input, the potential ways an object or scene can be interacted with by a robot. This is not just object detection; it's a functional understanding of physics and context.

• Examples: Predicting a mug's handle is graspable, a button is pressable, or a surface is pushable.
• Output: Typically a spatial map or heatmap over an image, highlighting regions where a specific skill (e.g., grasping) is feasible.
• Role in SayCan: The learned affordance model provides the probability p(can | state, skill) that a given low-level skill can be successfully executed in the current visual state, grounding the LLM's abstract plans.

Hierarchical task planning is the high-level 'Say' component of the paradigm. It refers to the decomposition of a complex, long-horizon natural language instruction into a sequence of executable sub-tasks or skills.

• Classic AI: Uses formal logic and symbolic planners (e.g., PDDL).
• LLM-based: Large Language Models act as the hierarchical planner, leveraging their knowledge of the world to propose logical step-by-step skill sequences.
• Role in SayCan: The LLM provides the probability p(say | instruction, skill) that a proposed skill is a good next step toward fulfilling the overall instruction. The final action is selected by maximizing the product p(say) * p(can).

A language-conditioned policy is a control function (often a neural network) that maps the current state of the environment and a natural language instruction directly to low-level robot actions. It is the executable realization of a planned skill.

• Function: π(actions | state, language_instruction).
• Training: Typically trained via imitation learning on datasets of (observation, language, action) triples or via reinforcement learning.
• Relation to SayCan: In the SayCan architecture, the library of feasible low-level skills is implemented as a set of pre-trained language-conditioned policies. The affordance model evaluates which of these policies is most likely to succeed.

RT-1 and RT-2 are transformer-based robot learning architectures from Google Robotics that represent a direct evolution beyond SayCan. They tokenize images, instructions, and actions into a single sequence.

• RT-1 (Robotics Transformer 1): A transformer model trained via imitation learning on a large-scale dataset of robot trajectories. It outputs actions directly, demonstrating strong generalization across tasks, objects, and environments.
• RT-2: A Vision-Language-Action (VLA) model. It co-fine-tunes a pre-trained vision-language model (like PaLI-X) on robot data, enabling the model to transfer web-scale knowledge to robotic control. It can perform chain-of-thought reasoning in a tokenized latent space before outputting actions.
• Contrast with SayCan: While SayCan is a framework that combines separate LLM and affordance models, RT-1/2 are end-to-end trained models that internalize both planning and control.

PaLM-E is an embodied multimodal model from Google that integrates a large language model (PaLM) with a vision transformer (ViT), feeding visual and sensor data directly into the LLM's token stream.

• Mechanism: Images are encoded into a sequence of tokens that are interleaved with language tokens. The LLM processes this multimodal sequence to output text (for planning) or tokenized actions (for control).
• Capability: Can perform emergent multimodal reasoning, such as answering questions about a scene, generating plans, or outputting control sequences, all within a single model.
• Relation to SayCan: PaLM-E can be seen as a more integrated, monolithic version of the 'Say' component. It grounds language in visual context internally, whereas SayCan uses a separate affordance model for physical grounding. PaLM-E was used as the planner in later SayCan experiments.

Embodied datasets are the large-scale collections of real-world robot interaction data required to train models like the affordance functions and policies used in SayCan and its successors.

• Purpose: Provide (observation, action, language) data for imitation learning.
• Key Examples:
  • RT-1 Dataset: ~130k episodes from 13 robots over 17 months.
  • Open X-Embodiment Dataset: A massive cross-embodiment dataset pooling data from 22 robot types, used to train RT-2-X.
  • Bridge Dataset: Focused on visuomotor manipulation across varied environments.
• Importance: The scale and diversity of these datasets are critical for moving from narrow skill libraries (as in early SayCan) to generalist, cross-embodiment models. They provide the 'Can' at scale.