Unified Tokenization

The core output is a single, flat sequence of discrete tokens that a standard transformer can process. Each token is an integer ID from a shared vocabulary. For example:

• An image might be tokenized into 256 patch tokens.
• A sentence might become 20 wordpiece tokens.
• These 276 tokens are concatenated into one sequence, often with special modality tokens (e.g., [IMG], [TXT]) prepended to indicate the data type of each segment.

After tokenization, each token ID is mapped to a dense vector via a shared embedding matrix. This projects all modalities into a common, high-dimensional vector space. A visual patch token and a text word token with similar semantic meaning are positioned close together in this space. This alignment is learned during pre-training using objectives like contrastive loss or masked token modeling.

While creating a flat sequence, the method must preserve crucial structural information.

• Images: A 2D grid of patches is often flattened using a spatial scanning order (e.g., raster scan), with learned positional embeddings added to inform the model of the original 2D relationships.
• Video: Frame patches are sequenced with additional temporal positional embeddings.
• Text: Standard sequential order is maintained with 1D positional encodings. The transformer's self-attention mechanism then dynamically re-learns these spatial and temporal relationships.

The unified tokenizer must handle a massive and diverse set of concepts. Common strategies include:

• Codebook-based Vocabularies: Using a VQ-VAE to quantize continuous visual features into discrete codes.
• Multi-Modal SentencePiece: Extending subword tokenization (e.g., BPE) to jointly learn merges across text and visual feature clusters.
• Hybrid Approach: Using a text-only vocabulary for language and a separate, learned visual codebook, with both mapped into the shared embedding space. The total vocabulary size can exceed 100k tokens.

Once tokenized into a unified sequence, a single transformer stack processes all tokens. This allows cross-modal attention, where a text token (e.g., "red") can directly attend to and incorporate information from all relevant image patch tokens. This seamless interaction is the key mechanism behind tasks like visual question answering and image captioning, as the model performs fusion at every layer.

Unified tokenization enables autoregressive generation across modalities. Models like Flamingo or GATO treat the next token prediction as a unified task, whether the next token is a text word or an image patch. This allows for:

• Image-Conditioned Text Generation: Generating a caption given image tokens.
• Text-Conditioned Image Generation: Predicting the next image patch tokens to form a coherent picture (as in Parti).
• Sequential Decision Making: Representing robotic actions as discrete tokens in the same sequence.

The core neural mechanism that enables a unified transformer to relate tokens from different modalities. After tokenization, attention scores are computed between all tokens in the sequence, allowing a text token (e.g., 'red') to directly attend to relevant visual patch tokens.

• Mechanism: Enables the model to learn which image regions are relevant to which words.
• Example: In VQA for 'What color is the car?', the 'color' and 'car' text tokens attend strongly to the visual tokens representing the vehicle's region.

A common, high-dimensional vector space where tokens from all modalities are projected. Unified tokenization is a prerequisite, converting raw inputs into discrete indices that are then mapped into this space via embedding layers.

• Purpose: Allows semantically similar concepts (e.g., the word 'dog' and an image of a dog) to have proximate vector representations.
• Critical for: Cross-modal retrieval, contrastive learning, and enabling a single transformer to process everything.

A single transformer stack that processes the unified token sequence. It treats all tokens identically, regardless of origin. This is the component that follows the modality-specific encoders (e.g., ViT, BPE tokenizer) and the linear projection into the shared space.

• Function: Applies self-attention and feed-forward layers across the mixed sequence.
• Benefit: Enables deep, late-interaction fusion where high-level textual and visual concepts can influence each other throughout the network's depth.

A key training objective used to align modalities in the shared embedding space. It does not directly perform tokenization but trains the projections that follow it.

• Mechanism: For a batch of image-text pairs, it maximizes similarity for correct pairs (positive) and minimizes similarity for incorrect pairings (negatives).
• Effect: Forces the model to learn that the unified tokens for a photo of a sunset and the caption 'beautiful sunset' should map to nearby vectors.

The overarching architecture that employs unified tokenization. Models like Flamingo, GATO, and RT-2 are canonical examples. Their design is defined by the ingestion of a heterogeneous token stream.

• Core Innovation: A single set of transformer weights processes vision, language, and sometimes action tokens.
• Scaling: Performance improves predictably with model size and dataset scale, following similar laws to pure language models.

The broader category of techniques for combining multimodal data. Unified tokenization is a specific, increasingly dominant strategy for intermediate fusion.

• Alternative Strategies:
  • Early Fusion: Combine raw pixels and text characters (rare).
  • Late Fusion: Run separate vision and language models, fuse only final outputs.
• Advantage of Unified Tokenization: Enables fine-grained, scalable fusion within a single transformer backbone, superior for generative and reasoning tasks.