Feature Fusion

Early fusion concatenates raw or low-level features from different modalities before feeding them into a primary model. This approach assumes tight, low-level correlation between modalities.

• Process: Features (e.g., pixel values, MFCCs, token IDs) are combined into a single input vector.
• Use Case: Ideal for tightly synchronized data, like video-audio streams, where joint low-level processing is beneficial.
• Challenge: Requires all modalities to be present at inference, making it less flexible for missing data.

Late fusion processes each modality through separate, dedicated models and combines their final outputs or decisions (e.g., logits, predictions).

• Process: Each modality has its own model pathway; results are aggregated via averaging, voting, or a meta-learner.
• Use Case: Effective for modular systems or when modalities are processed asynchronously. Common in ensemble methods.
• Advantage: Robust to missing modalities; individual model pathways can be updated independently.

Hybrid fusion combines elements of early and late fusion, often using intermediate representations. It allows for both low-level interaction and high-level decision combining.

• Process: Modalities may be fused at multiple stages within a network architecture.
• Architecture Example: A model might use early fusion for visual and textual features, then later fuse the result with a separate audio model's output.
• Benefit: Balances the representational power of early fusion with the flexibility of late fusion.

Attention-based fusion uses mechanisms like cross-attention to dynamically weight and integrate features from different modalities based on their contextual relevance.

• Core Mechanism: A sequence of queries from one modality attends to keys and values from another, allowing the model to focus on the most informative cross-modal signals.
• Model Example: Central to architectures like Flamingo and Stable Diffusion, where text queries attend to visual latents.
• Advantage: Creates context-aware, non-linear combinations superior to simple concatenation or averaging.

Tensor fusion models high-order interactions between modalities by computing the outer product of their feature vectors, creating a comprehensive multi-dimensional representation.

• Process: For feature vectors v1, v2, their outer product creates a matrix capturing all multiplicative interactions.
• Representation Power: Can theoretically model all possible unimodal, bimodal, and trimodal interactions.
• Computational Cost: The fused tensor grows exponentially with the number of modalities, often requiring factorization techniques (e.g., Tucker decomposition) for practicality.

Gated fusion employs learnable gating mechanisms (inspired by LSTMs or GRUs) to control the flow of information from each modality into the fused representation.

• Process: A gating network outputs a set of weights (between 0 and 1) that modulate the contribution of each modality's feature vector.
• Function: Allows the model to learn when to trust or ignore specific modalities based on the input context or task.
• Application: Useful in noisy environments or for tasks where the relevance of modalities varies significantly.

A mechanism in transformer architectures where a sequence of queries from one modality (e.g., text) attends to a sequence of keys and values from another (e.g., image patches). This allows the model to dynamically retrieve and fuse relevant information across modalities.

• Core Function: Enables fine-grained, context-aware fusion by computing attention weights between every element in two sequences.
• Key Use Case: Central to models like Flamingo and Stable Diffusion, where text tokens attend to visual features to guide generation or reasoning.

A self-supervised learning paradigm that trains a model to pull positive pairs of data points (e.g., an image and its caption) closer together in an embedding space while pushing negative pairs apart. This is foundational for creating aligned, shared representations.

• Primary Objective: Maximize the mutual information between paired modalities.
• Common Loss: InfoNCE Loss is the standard objective function for this task.
• Famous Model: CLIP (Contrastive Language-Image Pre-training) is trained this way, creating a unified text-image embedding space.

A common, lower-dimensional vector space where representations from multiple modalities are encoded. This enables direct comparison, retrieval, and reasoning across different data types.

• Engineering Goal: Achieve modality alignment so that semantically similar concepts (e.g., "dog") have similar vectors, whether from text, image, or audio.
• Enabling Technology: Created via projection layers that map different modalities into this unified space.
• Critical For: Cross-modal retrieval, translation, and any downstream task requiring joint understanding.

An approach where a single, shared model architecture processes and represents data from various input types, abstracting away the specifics of the original modality.

• Architecture Example: The Perceiver IO model uses a universal transformer backbone that first projects any input (images, audio, text) into a latent bottleneck for processing.
• Advantage: Simplifies system design by using one model for all inputs, rather than separate encoders per modality.
• Challenge: Requires careful design to handle vastly different data structures and information densities.

A family of techniques that use attention mechanisms to integrate multimodal features. Unlike simple concatenation, it dynamically weights and combines information based on its computed relevance to the task or context.

• Methods Include: Cross-attention, co-attention, and self-attention on concatenated multimodal tokens.
• Dynamic Weighting: Allows the model to focus on the most salient visual feature when answering a specific text question, for instance.
• Superiority: Generally outperforms early (input-level) or late (decision-level) fusion by enabling richer, mid-level interactions.

The process of training a foundational model on large-scale datasets containing multiple data types (e.g., billions of image-text pairs). This teaches the model general-purpose, aligned representations transferable to many downstream tasks.

• Objective: Learn a robust cross-modal embedding space and fusion capabilities from vast, weakly supervised data.
• Outcome: Produces models like CLIP, Flamingo, and GPT-4V that exhibit emergent multimodal reasoning.
• Transfer Learning: These pre-trained models are then fine-tuned with adapter layers or LoRA for specific applications like VQA or medical imaging.