Late Fusion

An integration strategy where raw features from different modalities are combined at the initial input layer before any significant processing. This allows the model to learn joint representations from the ground up, capturing fine-grained cross-modal interactions immediately.

• Mechanism: Raw signals (e.g., pixel values and word embeddings) are concatenated or gated at the input level.
• Trade-off: High computational cost and requires perfectly synchronized, aligned data.
• Contrast: Unlike late fusion, there is no independent unimodal processing; a single model learns everything jointly.

A hybrid approach where modality-specific encoders process inputs independently for several layers before cross-attention mechanisms allow features to intermix at multiple intermediate network depths. This enables the model to learn both unimodal abstractions and complex cross-modal relationships.

• Architecture: Typically uses separate initial encoders with shared transformer layers at the bottleneck.
• Benefit: Balances the unimodal specialization of late fusion with the rich interaction of early fusion.
• Example: The ViLBERT and LXMERT architectures use co-attentional transformer layers to fuse text and visual features mid-network.

The core neural operation enabling intermediate fusion, where queries from one modality attend to keys and values from another. This allows a text token to dynamically retrieve relevant visual regions, or an image patch to gather semantic context from a caption.

• Operation: Attention(Q_text, K_image, V_image) fuses text into vision, and vice versa.
• Role in Late Fusion: Absent in pure late fusion, which is why cross-modal interactions are only learned at the final decision layer.
• Scalability: Cross-attention scales quadratically with sequence length, making it computationally expensive for long sequences.

A specialized neural network that transforms raw input from a single modality into a dense, fixed-dimensional feature representation. In a late fusion pipeline, these encoders operate entirely independently and are often pre-trained on unimodal tasks.

• Text Encoder: Typically a transformer like BERT or T5 that outputs a contextualized embedding.
• Vision Encoder: Often a Vision Transformer (ViT) or CNN that outputs a global image descriptor or patch features.
• Independence: Because encoders do not share weights or gradients during training in late fusion, they can be swapped or updated modularly without retraining the entire system.

A shared high-dimensional vector space where representations from different modalities are projected to enable direct similarity comparison. Late fusion often relies on a final projection layer that maps independently computed unimodal embeddings into this common space before fusion.

• Training Objective: Contrastive loss functions like InfoNCE align matched pairs (e.g., an image and its caption) while pushing apart non-matched pairs.
• Result: Cosine similarity between a text vector and an image vector in this space directly measures semantic relatedness.
• Example: CLIP's joint embedding space allows zero-shot image classification by comparing an image embedding to text embeddings of class labels.

Techniques designed to reduce the generation of text that is factually inconsistent with or unsupported by the provided visual input. Late fusion architectures are particularly susceptible to hallucination because the language model's prior knowledge can override weak visual signals fused only at the final layer.

• Causal Factor: When visual features are compressed into a single vector late in the pipeline, the generative decoder may default to its strong language priors.
• Mitigations: Include up-weighting visual attention, using contrastive decoding to penalize text-only hypotheses, and injecting visual grounding signals earlier in the generation process.
• Relevance: This is a key failure mode that motivates moving from pure late fusion toward intermediate fusion for high-stakes VQA tasks.