Multi-Modal Transformer

The first step in a multi-modal transformer is converting raw inputs from different modalities into a common sequence of tokens. This involves:

• Text: Tokenized via a subword vocabulary (e.g., BPE).
• Images: Split into fixed-size patches (e.g., 16x16 pixels) and linearly projected into patch embeddings.
• Audio: Converted into spectrograms and then tokenized into patch-like segments.
• Modality-specific tokens (e.g., [IMG], [AUD]) are prepended to each sequence to inform the model of the input type. This unified token stream enables a single transformer backbone to process everything.

This is the core mechanism enabling modalities to interact. Unlike standard self-attention, cross-modal attention allows tokens from one modality to attend to and influence tokens from another.

• Key Implementation: In an encoder-decoder setup, a text decoder can use cross-attention to attend to encoded image tokens. In a single-stack encoder, modalities are interleaved, allowing self-attention to operate across them.
• Function: It allows a model to, for example, focus on specific image regions when generating a textual description, or to resolve textual ambiguity by referencing visual context.

Before fusion, raw inputs are often processed by specialized, pre-trained encoders to extract high-quality features.

• Common Components:
  • Text: A pre-trained language model (e.g., BERT, T5 encoder).
  • Vision: A pre-trained vision transformer (ViT) or convolutional neural network (CNN).
  • Audio: A pre-trained audio spectrogram transformer (AST).
• These encoders convert raw data into a sequence of feature vectors (tokens). Their parameters may be frozen or fine-tuned during multi-modal training, balancing prior knowledge with task-specific adaptation.

The architectural goal is to project all modality-specific representations into a shared semantic vector space.

• Purpose: Enables direct comparison and operations across modalities (e.g., measuring similarity between a text query and an image).
• How it's Achieved: Through contrastive pre-training (e.g., CLIP) or via the transformer's attention mechanism itself, which learns to align representations during tasks like image captioning or VQA. The final hidden states of the transformer serve as the aligned embeddings.

Defines how and when information from different streams is combined. Key strategies include:

• Early Fusion: Raw tokens from all modalities are concatenated at the input and processed by a single transformer. Simple but can be inefficient.
• Late Fusion: Each modality is processed independently by separate transformers, and their final representations are combined (e.g., concatenated, averaged) for a downstream task. Less interaction.
• Hierarchical Fusion: A hybrid approach where modalities interact at multiple layers of the network, allowing for gradual, sophisticated integration. This is common in state-of-the-art architectures.

Multi-modal transformers are implemented in several distinct model families:

• Encoder-Only (e.g., ViLBERT, LXMERT): Use dual-stream encoders with cross-modal attention layers for understanding tasks like VQA.
• Encoder-Decoder (e.g., Flamingo, GIT): A visual encoder feeds into a text decoder via cross-attention, optimized for generative tasks like captioning.
• Single-Stack (e.g., Unified-IO, OFA): A pure transformer that treats all tasks (vision, language, generation) as sequence-to-sequence problems, using a unified vocabulary of discrete tokens.

In a single-stream or merged architecture, tokens from all input modalities are concatenated into a single, unified sequence before being processed by a shared transformer encoder. A single set of transformer layers attends across all tokens, regardless of modality.

• Key Feature: Enables deep, early fusion of information across modalities within the attention mechanism itself.
• Example: Vision Transformer (ViT) for images can be extended by prepending text tokens with a [CLS] token; the model learns cross-modal relationships through self-attention.
• Trade-off: Highly expressive but requires the model to learn modality-agnostic representations from the start, which can be data-inefficient.

A dual-stream architecture processes each input modality through separate, modality-specific transformer encoders. The streams are fused at a later stage, typically via a cross-modal attention layer or a fusion network.

• Key Feature: Allows for specialized, unimodal feature extraction before cross-modal interaction.
• Example: CLIP (Contrastive Language-Image Pre-training) uses separate image and text encoders. Fusion occurs via a contrastive loss that aligns their output embeddings in a joint space, not via attention.
• Trade-off: More modular and often easier to pre-train on unimodal data, but cross-modal understanding is shallower than in single-stream designs.

This advanced pattern uses N modality-specific encoders (e.g., for video, audio, text) whose outputs are fused through dedicated cross-attention modules. Each modality can attend to the others in a structured way.

• Key Feature: Enables flexible, query-driven fusion. For example, the language stream can "attend to" specific regions of the visual stream to answer a question.
• Example: Flamingo and other vision-language models (VLMs) use a pretrained language model (LM) as one stream and a visual encoder as another. Perceiver resampler tokens from the visual encoder are fed as context into the LM using cross-attention layers interleaved in the LM.
• Use Case: Ideal for tasks requiring complex, asymmetric reasoning between modalities, like Visual Question Answering (VQA).

The core engineering challenge is converting raw, heterogeneous data (pixels, waveforms, characters) into a unified sequence of tokens that a transformer can process. This involves:

• Images: Split into fixed-size patches (e.g., 16x16 pixels), linearly projected to token embeddings.
• Audio: Convert spectrograms into patch-like segments ("audio patches").
• Video: Treat as a sequence of image frames, each tokenized into patches, often with added temporal positional embeddings.
• Text: Tokenized via subword methods (e.g., SentencePiece, WordPiece).

A modality-type embedding is typically added to each token to inform the model of its source (e.g., [IMAGE], [TEXT]).

A dominant pre-training objective for dual-stream architectures. The model is trained on massive datasets of paired data (e.g., images and captions).

• Mechanism: A contrastive loss (e.g., InfoNCE) is used to maximize the similarity (cosine) between the embeddings of matched pairs (positive samples) and minimize it for unmatched pairs (negative samples) in a batch.
• Outcome: Creates a joint embedding space where semantically similar concepts across modalities (e.g., a dog picture and the text "a dog") have similar vector representations.
• Power: Enables zero-shot transfer. After pre-training, the model can perform novel tasks like image classification by comparing an image's embedding to text embeddings of class labels (e.g., "a photo of a dog").

Models like GPT-4V and Flamingo are trained with a causal language modeling objective (next-token prediction) on interleaved sequences of text and visual tokens.

• Mechanism: Given a sequence of image tokens and text tokens, the model is trained to predict the next text token in an autoregressive fashion. The image tokens provide contextual conditioning.
• Outcome: The model learns to generate text conditioned on visual (and other) inputs, enabling open-ended cross-modal generation and dialogue.
• Capability: This is the foundation for multi-modal chatbots and assistants that can discuss image content, generate stories from pictures, or answer complex questions about charts and diagrams.

A specialized class of multi-modal model that jointly processes visual inputs (images, video frames) and textual inputs. VLMs are a primary application domain for multi-modal transformers, enabling tasks like:

• Visual Question Answering (VQA): Answering text questions about image content.
• Image Captioning: Generating descriptive text for visual scenes.
• Visual Grounding: Linking phrases in text to specific regions in an image. They are often built using a dual-encoder architecture (separate vision and text encoders) or a fusion encoder architecture, with the multi-modal transformer serving as the core fusion mechanism.

The core technique of combining information from two or more different data modalities (e.g., vision, language, audio) to create a unified, comprehensive representation. Multi-modal transformers implement sophisticated fusion strategies:

• Early Fusion: Raw or lightly processed inputs are concatenated before the transformer layers.
• Late Fusion: Each modality is processed independently by separate encoders, and their outputs are combined at a high level.
• Cross-Modal Attention: A middle-fusion approach where transformer attention layers compute interactions between tokens of different modalities, allowing deep, bidirectional integration. This is the defining mechanism of advanced multi-modal transformers.

The process of learning a shared semantic space where representations from different modalities are positioned such that semantically similar concepts are close together. This is the foundational goal that enables downstream tasks like cross-modal retrieval. Key methods include:

• Contrastive Learning: Used by models like CLIP.
• Cross-Modal Attention: Allows tokens from one modality to directly attend to and influence representations in another.
• Triplet Loss: A learning objective that enforces relative distances between anchor, positive, and negative examples across modalities. The modality gap—the inherent distributional mismatch between modalities—is the primary challenge alignment techniques must overcome.

A unified vector space where representations (embeddings) from different modalities are projected. This is the mathematical result of successful cross-modal alignment. Its properties enable critical operations:

• Cross-Modal Retrieval: Finding an image via a text query (and vice-versa) by measuring vector similarity (e.g., cosine similarity).
• Zero-Shot Classification: Classifying an image by comparing its embedding to text embeddings of class labels.
• Arithmetic Operations: Performing analogical reasoning (e.g., "king - man + woman") across modalities. The space is typically high-dimensional (e.g., 512 or 768 dimensions) and learned via deep neural networks.

A self-supervised learning paradigm crucial for training multi-modal transformers without exhaustive labeled data. It teaches a model to distinguish between similar and dissimilar data pairs:

• Positive Pairs: Data points that are semantically aligned (e.g., an image and its accurate caption).
• Negative Pairs: Data points that are not aligned (e.g., a random image and caption).
• Objective Function: The model learns to pull positive pairs closer in the embedding space and push negative pairs apart. Common loss functions include InfoNCE (Noise-Contrastive Estimation). This technique is the engine behind models like CLIP and is fundamental for learning robust joint embedding spaces.