Multimodal Transformer

The core architectural innovation enabling multimodal interaction. Cross-attention allows the model to compute attention scores between sequences from different modalities. For example, when processing the word 'dog' in a text caption, the model can use cross-attention to dynamically focus on the corresponding visual region in a paired image. This mechanism replaces the self-attention used within a single modality, creating a bridge for information to flow between text tokens, image patches, or audio frames.

Before fusion, raw data from each input type is processed by specialized encoders. These are often pre-trained models adapted to extract meaningful features:

• Text: A tokenizer and language model backbone (e.g., BERT, T5).
• Vision: A convolutional neural network or Vision Transformer (ViT) to split images into patches.
• Audio: A waveform encoder or spectrogram-based model (e.g., HuBERT, Wav2Vec2). These encoders convert heterogeneous inputs into a sequence of token embeddings, providing a unified, high-level representation for the transformer layers.

Defines where and how information from different modalities is combined within the transformer stack.

• Early Fusion: Modality embeddings are concatenated at the input. The transformer processes the mixed sequence from the first layer.
• Late Fusion: Each modality is processed by separate transformer stacks; their final outputs are combined (e.g., averaged) for the task.
• Intermediate Fusion: The most common approach for multimodal transformers. Modalities are processed independently for several layers, then fused via cross-attention at a specific intermediate layer, allowing for deeper, more flexible interaction.

Standard transformers use positional encodings to inject order information. In a multimodal context, this must account for the structure of concatenated sequences. Strategies include:

• Absolute Positioning: Treating the full concatenated sequence (e.g., [IMG_PATCH_1, ..., IMG_PATCH_N, TEXT_TOKEN_1, ...]) as one long sequence.
• Modality-Aware Encoding: Adding a learnable modality type embedding (e.g., [VISION], [TEXT]) to each token, in addition to its position embedding, to inform the model of the token's origin.
• Relative Positioning: Used in video or audio-text models to handle temporal alignment across modalities.

Multimodal transformers are typically pre-trained on massive, unlabeled datasets using self-supervised tasks that force the model to learn alignment. Key objectives include:

• Masked Modeling: Randomly masking tokens or patches in one modality and training the model to reconstruct them using context from both modalities.
• Contrastive Learning (e.g., CLIP): Using a dual-encoder architecture to pull matching image-text pairs closer in a joint embedding space while pushing non-matching pairs apart, often optimized with an InfoNCE loss.
• Cross-Modal Matching: A binary classification task to predict if a given pair of modalities (e.g., an image and a caption) are correctly aligned.

The final component determines the model's capability for downstream applications. A single pre-trained backbone can be adapted via different output heads:

• Classification Head: For tasks like visual question answering or sentiment analysis from video+audio.
• Sequence Generation Head: For modality translation, such as image captioning (vision-to-text) or text-to-image generation.
• Retrieval Head: For cross-modal retrieval, using the pooled embeddings from the joint space to find similar items across modalities.
• Regression Head: For dense prediction tasks like visual grounding, predicting bounding box coordinates from text queries.

These models process image-text pairs for tasks like visual question answering (VQA) and image captioning. They typically use a vision encoder (like ViT or CNN) to create image patch embeddings and a text encoder (like BERT) for token embeddings, fused via cross-attention.

• Examples: LXMERT, VisualBERT, ViLT.
• Fusion Strategy: Primarily intermediate fusion, where cross-attention layers allow text tokens to attend to image patches and vice versa.
• Key Innovation: ViLT simplified the architecture by using a linear projection for image patches, eliminating heavy convolutional feature extractors for faster training.

This highly scalable architecture uses two separate, parallel encoders—one for each modality—trained with a contrastive loss (like InfoNCE) to align representations in a joint embedding space.

• Core Mechanism: Image and text are encoded independently; similarity is computed in the shared space. Enables efficient cross-modal retrieval.
• Advantage: Encoders are decoupled, allowing for pre-computation of embeddings and extremely fast retrieval at scale.
• Application: Powers zero-shot classification and is the backbone for image generation models like DALL-E 2 and Stable Diffusion via text-guided search.

These generative models are built on a encoder-decoder transformer framework. They ingest multiple interleaved sequences of visual and textual data and generate free-form textual outputs.

• Architecture: A perceiver resampler or similar module condenses variable-length visual sequences into a fixed set of tokens, which are fed into a large language model decoder alongside text tokens.
• Capability: Few-shot in-context learning across vision-language tasks without task-specific fine-tuning.
• Use Case: Multimodal dialogue, long-form video description, and complex reasoning requiring integration of visual context and prior knowledge.

Designed to process synchronized video and audio waveforms, these models perform tasks like automatic speech recognition (ASR) with visual lip-reading, audio-visual event classification, and sound source localization.

• Input Processing: Uses 3D CNNs or ViT for video frames and 1D convolutional layers or audio spectrogram transformers for audio.
• Fusion Challenge: Requires temporal alignment of audio and visual features. Often uses cross-modal attention or sensor fusion at intermediate layers.
• Example Models: AV-HuBERT (for self-supervised speech recognition), Perception Test models.

These models frame all tasks—across modalities—as a unified sequence-to-sequence problem. They convert images, text, and sometimes other data into a common vocabulary of discrete tokens.

• Methodology: Images are tokenized into visual words (e.g., using VQ-VAE). The model is then trained with a mixture of denoising objectives on interleaved sequences of these tokens.
• Goal: Generalist multimodal capability from a single model, handling tasks from image captioning to text-to-image generation via the same forward pass.
• Significance: Represents a move towards foundation models for multimodal AI, reducing the need for task-specific architectures.

These are vision-language-action (VLA) transformers that process camera inputs and natural language instructions to output low-level robotic actions (e.g., joint torques, gripper commands).

• Core Function: Close the perception-action loop. They translate high-level goals ("pick up the blue block") into executable motor controls.
• Architecture: Often built upon large vision-language models, with an additional action head that decodes the transformer's output into a continuous action space.
• Training: Uses large-scale datasets of robotic trajectories paired with natural language descriptions, enabling in-context generalization to new tasks.