Flamingo Architecture

The core innovation of Flamingo is the gated cross-attention layer (Xattn-d). This mechanism allows the language model to condition its text generation on visual features. It works by:

• Projecting visual features from a frozen vision encoder into keys and values.
• Using the language model's hidden states as queries.
• Applying a sigmoid gating mechanism to control the influence of the visual stream, allowing the model to learn when to attend to visual information.
• This design enables the integration of a pre-trained, frozen language model (like Chinchilla) with visual inputs without catastrophic forgetting of linguistic capabilities.

The Perceiver Resampler is a module that processes a variable number of visual feature vectors (from images or video frames) into a fixed number of visual tokens. This is critical because:

• Vision encoders (like NFNet) output a large, variable-sized grid of features per image.
• The resampler uses cross-attention with a set of learnable latent queries to condense this information.
• It produces a manageable, fixed-length sequence of visual tokens that can be efficiently fed into the subsequent gated cross-attention layers.
• This allows Flamingo to handle multiple images or video frames within a single context window.

Flamingo leverages large, frozen pre-trained models for both vision and language, avoiding the cost of end-to-end multimodal training from scratch.

• Vision Encoder: A CLIP ViT or NFNet model, pre-trained on web-scale image-text data, is used to extract visual features. Its weights remain frozen.
• Language Model: A Chinchilla-scale decoder-only transformer, pre-trained on text, serves as the core reasoning engine. Its weights are also frozen.
• Only the newly added components (Perceiver Resampler, gated cross-attention layers) are trained from scratch on multimodal data. This is a form of parameter-efficient adaptation.

Flamingo is designed to process interleaved sequences of text, images, and videos within a single context. This is facilitated by:

• Special <image> and <video> tokens that are inserted into the text sequence to mark the location of visual data.
• When the language model processes one of these tokens, the corresponding visual features (processed by the Perceiver Resampler) are made available to the subsequent gated cross-attention layers.
• This allows for few-shot in-context learning where examples can be provided as image-text or video-text pairs within the prompt itself, mimicking the model's training data format.

Flamingo is trained on massive, diverse datasets of interleaved image/video and text, using a causal language modeling objective.

• Primary Objective: Standard autoregressive next-token prediction on text, conditioned on any preceding visual and textual context.
• Training Data: A mixture of:
  • M3W: A curated dataset of web pages with interleaved images and text.
  • ALT-Text: Billions of image-text pairs from the web.
  • VTP: Video-text pairs.
• The model never sees a loss on the visual features themselves; learning is driven entirely by the text prediction task, which forces the model to ground language in vision.

Flamingo established a highly influential paradigm for building powerful multimodal systems.

• Few-Shot Pioneer: It demonstrated that large models could perform few-shot learning on novel multimodal tasks without task-specific fine-tuning.
• Frozen Backbone Strategy: Proved the efficacy of composing large, frozen unimodal models with lightweight trainable adapters, a pattern followed by later models like BLIP-2 and Fuyu-8B.
• Bridge to VLMs: Directly inspired the development of general-purpose Vision Language Models (VLMs) capable of open-ended dialogue about images.
• Its core mechanism—cross-attention as an adapter—remains a foundational technique for modality fusion.

A core mechanism in transformer architectures where a sequence of queries from one modality (e.g., language) attends to a sequence of keys and values from another modality (e.g., vision). This enables dynamic, context-aware fusion of information across disparate data sources and is the fundamental building block of Flamingo's gated cross-attention layers.

• Key Function: Allows the model to 'look' at relevant parts of an image when generating a text answer.
• Architectural Role: Forms the bridge between pre-trained, frozen vision encoders and language models in few-shot learning setups.

A self-supervised learning paradigm that trains a model to learn a shared embedding space by pulling representations of semantically similar data points (positive pairs) closer together while pushing dissimilar ones (negative pairs) apart. This is foundational for aligning modalities without explicit labels.

• Common Objective: Uses the InfoNCE loss to maximize mutual information between positive pairs.
• Example Model: CLIP (Contrastive Language-Image Pre-training) uses this technique to create a unified text-image space, a precursor concept to more integrated architectures like Flamingo.

A design principle for creating model components that can process and represent data from various input types (text, image, audio) using a single, shared architecture. The goal is to abstract away the specifics of the original data format.

• Architectural Examples: The Perceiver IO model uses a latent bottleneck and cross-attention to handle arbitrary modalities. Flamingo's use of a universal transformer backbone for processing visual tokens is an application of this principle.
• Engineering Benefit: Simplifies system design by reducing the need for modality-specific processing pipelines.

Small, trainable neural network modules inserted into a pre-trained, frozen model to adapt it for a new task or modality. This enables parameter-efficient fine-tuning, as only the adapter weights are updated, preserving the original model's knowledge.

• Use in Flamingo: While Flamingo uses gated cross-attention, the concept is similar: large pre-trained vision and language models are kept frozen, and only the connecting layers (the 'adapters' between modalities) are trained.
• Related Technique: LoRA (Low-Rank Adaptation) is a specific, popular method for implementing adapter-like functionality in large language models.

A canonical multimodal AI task where a model must answer a natural language question based on the content of an image. It requires deep joint understanding of visual scenes and linguistic queries.

• Flamingo's Purpose: Flamingo Architecture was explicitly designed to achieve strong few-shot performance on tasks like VQA without task-specific fine-tuning.
• Benchmark: Models are evaluated on datasets like VQAv2 and GQA, measuring their ability to reason about objects, attributes, relationships, and actions in images.

The process of training a neural network on large-scale datasets containing multiple data types (e.g., billions of image-text pairs scraped from the web). This teaches the model foundational, general-purpose representations that can be transferred to various downstream tasks.

• Foundation for Integration: Flamingo leverages models that have undergone massive vision-only and language-only pre-training. Its architecture is designed to efficiently combine these pre-trained components.
• Outcome: Produces models with emergent capabilities like in-context learning on multimodal prompts, where a few examples guide the model's response.