Multimodal Transformer

A Multimodal Transformer begins with separate, specialized encoders to convert raw data from each modality into a sequence of token embeddings. These encoders are tailored to the unique structure of their input.

• Text: Uses a standard tokenizer and embedding layer.
• Vision: Employs a patchification process, where an image is split into fixed-size patches, linearly projected into tokens.
• Audio: Often uses a convolutional neural network (CNN) or spectrogram transformer to extract frame-level features.

The output of each encoder is a sequence of tokens, which are then prepended with a special [CLS] or [MODALITY] token to aggregate global information for that modality.

The core fusion mechanism. After modality-specific encoding, token sequences are concatenated and fed into a stack of standard transformer layers. Within these layers, cross-attention (a key component of the self-attention mechanism) allows tokens from one modality to attend to tokens from all other modalities.

• This enables the model to establish semantic relationships across modalities, such as linking the word "dog" to visual patches containing a dog.
• Architectures vary: some use early fusion (concatenating all tokens at the input), while others use co-attention layers where modalities interact in a more structured, alternating fashion.

Through joint training on aligned multimodal data (e.g., image-text pairs), the transformer learns to project all input tokens into a unified semantic vector space. In this space, semantically similar concepts from different modalities are mapped to nearby locations.

• This is the foundational output enabling tasks like cross-modal retrieval (e.g., text-to-image search).
• The [CLS] token's final hidden state often serves as a global, fused representation of the entire multimodal input for downstream classification or regression tasks.

Many state-of-the-art models (e.g., CLIP, ALIGN) are pre-trained using a contrastive learning objective on massive datasets of paired data (e.g., billions of image-text pairs).

• The model learns by trying to correctly identify which text description matches a given image from a batch of negative examples.
• The loss function, typically InfoNCE, pushes the embeddings of matched pairs closer together while pushing non-matches apart in the unified space.
• This pre-training creates powerful, aligned representations without explicit category labels.

Beyond understanding, some Multimodal Transformers are architected for conditional generation. These are often decoder-only or encoder-decoder models.

• Image Generation from Text: Models like DALL-E and Stable Diffusion use a transformer to process text prompts and guide a diffusion model.
• Text Generation from Images: Models like Flamingo or GPT-4V interleave visual tokens with text tokens, allowing the language model to attend to visual context and generate descriptive or analytical text.
• This requires careful conditioning mechanisms, often using cross-attention layers where the text decoder attends to the encoded visual sequence.

Different designs prioritize efficiency, modality count, or task specificity.

• Dual-Encoder: Uses two separate towers (e.g., for text and image) that project into a shared space. Efficient for retrieval but lacks deep, token-level interaction. Example: CLIP.
• Fusion Encoder: A single transformer encoder processes concatenated tokens from all modalities, enabling rich cross-modal interaction. Example: ViLT.
• Encoder-Decoder: An encoder processes all input modalities, and a decoder generates a sequence (e.g., text) conditioned on this fused representation. Example: BLIP for image captioning.
• Mixture-of-Experts (MoE): Employs specialized sub-networks (experts) for different modalities or data regions, activated dynamically for scalable, efficient training.

The core fusion mechanism within a multimodal transformer. It allows tokens from one modality (e.g., text) to attend to and incorporate information from tokens of another modality (e.g., image patches). This is distinct from self-attention, which operates within a single sequence.

• Key Function: Enables the model to learn which parts of an image are relevant to a specific word, or which words describe a specific region of a video frame.
• Architectural Role: Typically implemented in transformer decoder layers or specialized fusion layers, forming the bridge between separate modality-specific encoders.

The unified, high-dimensional vector space where representations from different modalities are projected. The primary goal of a multimodal transformer is to learn this space.

• Semantic Proximity: In a well-learned joint space, the vector for the word "dog" and the vector for an image of a dog will have high cosine similarity.
• Enables Cross-Modal Tasks: This shared space is the foundation for cross-modal retrieval, zero-shot classification, and captioning, as comparisons are made directly between vectors regardless of origin modality.

A dominant self-supervised training paradigm for aligning multimodal data. It teaches the model by distinguishing between matched and unmatched pairs.

• Objective: Maximize the similarity (pull together) of embeddings for a positive pair (e.g., an image and its correct caption) while minimizing similarity (push apart) for negative pairs (e.g., that image with a random caption).
• Common Loss Functions: InfoNCE Loss and Triplet Loss are specifically designed for this objective. Training involves careful hard negative mining to improve discriminative power.

The specialized neural network component that converts raw data from a single modality into a sequence of token embeddings. A multimodal transformer typically employs one encoder per input type.

• Examples: A Vision Transformer (ViT) for images, a BERT-style encoder for text, or a 1D convolutional network for audio waveforms.
• Output: Each encoder produces a sequence of feature vectors (tokens) that preserve the structural information of the input (spatial for images, temporal for audio/sequence). These token sequences are the inputs to the cross-attention fusion layers.

A fundamental capability enabled by a trained multimodal transformer and its joint embedding space. It involves using a query from one modality to find relevant data from another.

• Primary Use Cases: Text-to-image search (e.g., DALL·E's prior), image-to-text (automatic caption retrieval), video-to-audio, and vice-versa.
• Mechanism: The query and the database items are all projected into the joint embedding space. Retrieval is performed using a nearest-neighbor search (e.g., based on cosine similarity) in this vector space.

The training objective and resulting state where the model's internal representations for the same concept, but from different modalities, correspond directly in the joint embedding space.

• The Core Challenge: Bridging the semantic gap between low-level perceptual features (pixels, sound frequencies) and high-level abstract meaning.
• Evaluation: Measured by the model's performance on tasks like zero-shot cross-modal transfer, where understanding learned from one modality (e.g., text descriptions of objects) is directly applied to another (e.g., classifying objects in images).