Multimodal Prompt

The raw data from each distinct sensory channel that constitutes the prompt. This is not a single stream but a collection of heterogeneous data types that must be individually encoded.

• Visual Inputs: Images, video frames, or 3D point clouds, often pre-processed into fixed-size patches.
• Linguistic Inputs: Natural language instructions, questions, or descriptive captions, tokenized into subword units.
• Other Modalities: Can include audio spectrograms, tactile sensor readings, or structured data (e.g., JSON), depending on the model's capabilities.

The model's modality-specific encoders (e.g., a Vision Transformer for images, a text tokenizer for language) transform these raw inputs into initial feature representations.

The process of converting disparate modality-specific features into a common, sequential format that a transformer-based architecture can process. This creates a joint input sequence.

• Visual Tokenization: Image patches are linearly projected into a sequence of patch embeddings, often with added positional encodings.
• Linguistic Tokenization: Text is converted into word or subword tokens with their own embeddings.
• Special Tokens: Tokens like [CLS], [SEP], or [IMG] are inserted to denote the start, separation, and type of different input segments.

Critical to this stage is temporal or spatial alignment, ensuring that the sequence order reflects the semantic relationship between modalities (e.g., a text token referring to an object is positioned near the corresponding image patch tokens).

The core neural mechanism within a multimodal transformer that enables information flow between modalities. After tokenization, the unified sequence is processed by transformer blocks where cross-modal attention layers compute attention scores between every token, regardless of its original modality.

• This allows a text token like "red ball" to directly attend to and incorporate features from the visual tokens representing the red ball in the image.
• The model builds a contextualized joint representation where each token's embedding is informed by all other tokens in the prompt.
• Architectures vary: some use early, dense cross-attention; others use late or gated fusion. The prompt structure must be compatible with the model's expected attention pattern.

The explicit natural language directive that defines the desired output format and task. This is often the primary textual component of the prompt and is crucial for zero-shot transfer.

• Explicit Instruction: "Generate a detailed caption for this image." or "Answer this question based on the video."
• Implicit Instruction: In conversational models, the instruction can be implied by the dialogue history and role (e.g., "You are a helpful assistant that describes images.").
• Output Formatting: Instructions often include specifications for structured output, such as "Return a JSON object with keys 'color' and 'count'." This leverages the model's pre-training on code and structured data.

This component steers the model's decoding head to generate text, regress coordinates, or produce other modality outputs.

Optional demonstrations included within the prompt itself to illustrate the desired input-output mapping without updating model weights. This is a form of prompt-based fine-tuning.

• Structure: Each example is a complete (multimodal input, target output) pair inserted before the final query.
• Purpose: Teaches the model task-specific patterns, output formatting, and reasoning chains. For example, showing an image with a question and the correct answer before presenting a new image-question pair.
• Considerations: Adding examples increases prompt length and computational cost. The selection and ordering of examples can significantly impact performance due to recency bias in the model's context window.

High-level directives that define the model's role, constraints, and behavioral guidelines. These are often prepended to the user-provided prompt and persist across a session.

• Role Definition: "You are a precise visual assistant for robotics."
• Safety & Guardrails: Instructions to avoid harmful content, refuse certain requests, or prioritize factual accuracy.
• Reasoning Constraints: Directives like "Think step by step" or "First, describe the scene, then answer the question" that can chain-of-thought reasoning.
• Context Management: Instructions on how to handle long contexts or prior conversation turns.

These meta-instructions shape the model's prior distribution over outputs before processing the specific multimodal query, acting as a high-level behavioral bias.

This is the most common form, where a visual input is paired with a textual directive. The model must ground the language in the visual scene to execute the command.

Examples:

• Visual Editing: "Take this photo of a street and make it look like a rainy night."
• Analysis & Description: "Describe the technical schematic in this image."
• Conditional Generation: "Generate a website mockup based on this hand-drawn wireframe sketch."

Used in embodied AI and robotics, this prompt provides a video demonstration paired with a high-level goal, instructing an agent to learn or replicate the observed behavior.

Examples:

• Imitation Learning: A video of a robot arm assembling a part, with the prompt: "Perform this assembly task."
• Procedural Understanding: A cooking video with the prompt: "List the steps required to make this dish."
• Trajectory Prediction: A dashcam video with the prompt: "Predict the next 5 seconds of movement for the cyclist."

This combines a sound clip with descriptive or instructional text, requiring the model to align auditory features with semantic meaning.

Examples:

• Transcription & Analysis: An audio clip of machinery with the prompt: "Transcribe the speech and identify any abnormal mechanical sounds."
• Conditional Audio Generation: A text description like "a busy coffee shop" with the prompt: "Generate ambient audio matching this scene."
• Emotion & Intent Recognition: A customer service call recording with the prompt: "Identify the customer's sentiment and key complaint."

Critical for autonomous systems, this prompt fuses spatial data (e.g., LiDAR point clouds, depth maps) with a natural language navigation instruction.

Examples:

• Embodied Navigation: A 3D scene representation with the prompt: "Go to the kitchen and find the blue mug on the counter."
• Manipulation Planning: A point cloud of a cluttered table with the prompt: "Grasp the screwdriver and place it in the toolbox."
• Spatial QA: A building floor plan with the prompt: "What is the most efficient route from the elevator to conference room B?"

This treats the visual layout and textual content of a document (PDF, scan, webpage) as a combined modality, enabling queries that require understanding format and content.

Examples:

• Form Extraction: An invoice scan with the prompt: "Extract the vendor name, total amount, and due date into a JSON schema."
• Diagram Reasoning: A flowchart image with the prompt: "Explain the decision logic starting from node 'A'."
• Multi-Document Synthesis: Several research paper PDFs with the prompt: "Compare the methodologies used in these three documents."

This prompt provides several images and asks the model to perform analysis, generation, or reasoning across the set, enforcing relational understanding.

Examples:

• Design Iteration: Three product concept sketches with the prompt: "Generate a new design that combines the ergonomics of image A with the style of image B."
• Anomaly Detection: A series of manufacturing line photos with the prompt: "Identify the image where the product is defective and describe the flaw."
• Story Continuation: A sequence of comic panels with the prompt: "Generate the next panel in the sequence."

A core neural mechanism enabling a model to dynamically focus on relevant parts of one modality based on context from another. For example, when processing the prompt "a red apple on the table," the text tokens for "red" and "apple" will attend to the corresponding visual regions in the accompanying image.

• Key Function: Computes attention scores between tokens from different data streams (e.g., text tokens to image patches).
• Architectural Role: The primary method for fine-grained interaction in transformer-based multimodal models like CLIP and Flamingo.

The process of converting raw inputs from distinct modalities into a common sequence of discrete tokens that can be processed by a single transformer model. This is a foundational step for handling multimodal prompts in a unified architecture.

• Standard Practice: Image patches are linearly projected into visual tokens, while text is split into subword tokens.
• Example: Models like GPT-4V and LLaVA use a vision encoder to convert an image into a sequence of tokens, which is then prepended to the text token sequence, forming one contiguous input stream for the language model.

A high-dimensional vector space where representations from different modalities are projected so that semantically similar concepts are close together, regardless of their original data type. This space is what makes cross-modal retrieval and comparison possible.

• Training Objective: Often learned using contrastive loss (e.g., CLIP's objective) to align image and text embeddings.
• Direct Application: Enables a model to understand that the vector for the word "dog" and the vector for a picture of a dog have high cosine similarity, grounding the language in visual concepts.

A multimodal architecture strategy where features from different modalities are integrated at one or more intermediate layers within a neural network. This allows for deep, iterative cross-modal interactions, which is more expressive than early or late fusion.

• Contrast with Other Strategies: More flexible than early fusion (combining raw inputs) and more interactive than late fusion (combining final outputs).
• Typical Implementation: Uses cross-modal attention layers interspersed throughout a transformer backbone to allow visual and linguistic features to influence each other repeatedly during processing.

A training objective that teaches a model to align multimodal representations by pulling positive pairs together and pushing negative pairs apart in a shared embedding space. InfoNCE loss is its most common, information-theoretic formulation.

• Mechanism: For a batch of image-text pairs, the loss maximizes the similarity score for the correct (positive) pair while minimizing scores for all incorrect (negative) pairings.
• Primary Use: The foundational pre-training objective for models like CLIP and ALIGN, which creates the aligned embedding space essential for zero-shot understanding of multimodal prompts.

A neural network component, typically a transformer decoder, that generates output sequences (like text or action tokens) based on a unified, modality-agnostic internal representation. It is the final stage that interprets the fused multimodal context from a prompt.

• Core Function: Takes the processed, fused representation of the multimodal prompt and autoregressively generates the appropriate output sequence.
• System Role: In a Vision-Language-Action model, this decoder might output a sequence of action tokens that are decoded into robot joint angles or navigation commands, all conditioned on the visual and linguistic prompt.