Multimodal BERT

This is the core mechanism enabling interaction between modalities. Unlike self-attention within a single modality, cross-attention allows the model to compute attention scores between elements of different sequences. For example, when processing an image-text pair, the text tokens can attend to relevant image patch embeddings, and vice-versa. This creates a dynamic, context-aware fusion of information, allowing the model to ground textual concepts in visual regions and understand visual scenes through language. Architectures like VisualBERT and ViLBERT implement this via transformer layers where queries come from one modality and keys/values from another.

Multimodal BERT models convert diverse inputs into a single, coherent sequence of tokens for the transformer. This involves:

• Text Tokenization: Using standard WordPiece or SentencePiece tokenizers.
• Visual Tokenization: Dividing an image into fixed-size patches (e.g., 16x16 pixels) and linearly projecting them into embeddings, often with added positional encodings.
• Special Tokens: Using unique tokens like [CLS] for aggregate representation, [SEP] to separate modalities, and [IMG] or [PATCH] to denote visual segments. All tokens are combined into one sequence, allowing the transformer to process them jointly from the start.

Models are often pre-trained using objectives that force alignment between modalities in a shared embedding space. A primary method is contrastive learning with an InfoNCE loss. The model is trained on large-scale noisy image-text pairs (e.g., from the web). The objective is to maximize the similarity between the embeddings of matching (positive) pairs while minimizing similarity for non-matching (negative) pairs. This teaches the model that the caption "a dog running" should be closer to an image of a running dog than to an image of a cat. This pre-training creates a powerful, aligned joint embedding space foundational for downstream tasks.

This extends BERT's classic Masked Language Modeling (MLM) to multiple modalities. The model must predict randomly masked tokens based on the context from both modalities.

• Masked Language Modeling: A text token is masked, and the model must predict it using surrounding text and the associated image.
• Masked Region Modeling (MRM): A visual region's embedding is masked, and the model must predict features of that region using other image patches and the accompanying text. This bidirectional training objective forces the model to build deep, cross-modal understanding and robust representations.

Multimodal BERT architectures employ different strategies for processing and fusing information:

• Single-Stream Architectures: Use one transformer stack that processes interleaved text and image tokens from the beginning (e.g., VisualBERT). Enables deep, early interaction.
• Dual-Stream Architectures: Use separate transformer encoders for each modality initially, followed by cross-attention layers for fusion (e.g., ViLBERT). Allows for modality-specific processing before interaction.
• Projection Heads: Small neural networks (often linear layers) map encoder outputs into a lower-dimensional joint embedding space where contrastive losses are applied.

The pre-trained model serves as a versatile foundation for a wide range of vision-language tasks through task-specific multimodal fine-tuning. The unified architecture allows it to be adapted by simply changing the final output layer. Common downstream applications include:

• Visual Question Answering (VQA): Answering natural language questions about an image.
• Image-Text Retrieval: Finding relevant images for a text query or vice-versa (cross-modal retrieval).
• Image Captioning: Generating descriptive text for an image.
• Natural Language for Visual Reasoning (NLVR): Determining if a textual statement is true about a set of images. The model's cross-attention mechanism is particularly effective for tasks requiring fine-grained cross-modal grounding.

A neural network mechanism that allows a model to compute attention scores between elements of different modalities. This enables the model to dynamically focus on relevant parts of one modality (e.g., a word) when processing another (e.g., an image region). It is the core architectural component in most multimodal transformers, including Multimodal BERT, for enabling deep interaction between data types.

A shared vector space where representations (embeddings) from different modalities are projected. The goal is for semantically similar concepts—like the word "dog" and an image of a dog—to have vectors that are close together. This space enables direct operations like similarity measurement and cross-modal retrieval. Models are often trained using contrastive learning to create this unified space.

A self-supervised learning paradigm used to train models to create effective joint embedding spaces. The model learns by being shown positive pairs (e.g., a matching image and caption) and negative pairs (non-matching data). It is trained to pull positive pairs closer and push negative pairs apart in the embedding space. A common loss function for this is the InfoNCE loss.

The overarching technique of combining information from two or more different data types (modalities) to produce a more robust and comprehensive representation for a downstream task. Key strategies include:

• Early Fusion: Combining raw or low-level features at the input.
• Late Fusion: Combining high-level outputs or decisions from separate models.
• Intermediate Fusion: Combining features at a middle network layer, allowing for interaction after some independent processing.

The practical task of searching for data in one modality using a query from a different modality. Common examples include:

• Text-to-Image: Finding relevant photos with a descriptive sentence.
• Image-to-Text: Retrieving captions or articles related to a given picture.
• Video-to-Audio: Finding soundtracks that match a video clip's mood. This task is a direct application of a well-aligned joint embedding space.

The two primary axes of cross-modal alignment:

• Semantic Alignment: Ensuring representations from different modalities correspond to the same high-level concept or meaning (e.g., aligning the word "running" with video frames of a person running).
• Temporal Alignment: Synchronizing sequences of data from different modalities along a timeline (e.g., aligning spoken words in an audio track with corresponding lip movements in a video, often using algorithms like Dynamic Time Warping).