Multimodal Autoregressive Generation

The core mechanism treats text tokens and visual patches as elements of a single, shared vocabulary. The model is trained to predict the next element in a sequence, regardless of its modality. This allows a single transformer to learn the joint distribution over text and images without separate encoders or decoders for each modality.

• Text tokens: Standard subword units from a sentencepiece tokenizer
• Visual tokens: Flattened, quantized image patches processed by a VQ-VAE or similar discrete autoencoder
• Training objective: Standard cross-entropy loss on the next token, applied uniformly across modalities

Before an image can be treated as a sequence, it must be converted into discrete tokens. A Vector Quantized Variational Autoencoder (VQ-VAE) or VQ-GAN compresses an image into a grid of latent codes from a learned codebook. Each code is an integer index, analogous to a word in a text vocabulary.

• Codebook size: Typically 8,192 or 16,384 discrete codes
• Compression ratio: A 256x256 image might be represented by a 16x16 grid of tokens
• Reconstruction: The decoder reconstructs the image from the token grid, ensuring the codes retain visual fidelity

The model can generate sequences where text and images alternate naturally, such as a document with embedded figures. The autoregressive process generates a token; if the token is an image-start marker, the model generates visual tokens until an image-end marker is produced, then resumes text.

• Control: Special control tokens signal modality switches
• Example: A prompt like 'Describe the following image: [IMG]' causes the model to generate a text description, then generate the image tokens, then continue with more text
• Applications: Interactive tutorials, illustrated stories, technical documentation with generated diagrams

To improve the quality and text-alignment of generated images, classifier-free guidance (CFG) is applied during inference. The model performs two forward passes: one conditional on the text prompt and one unconditional. The final logits are a weighted combination that amplifies the influence of the conditioning signal.

• Guidance scale: A hyperparameter (typically 3.0–7.0) controlling the trade-off between fidelity and diversity
• Mechanism: logits_guided = logits_uncond + scale * (logits_cond - logits_uncond)
• Effect: Produces sharper, more prompt-adherent images at the cost of reduced diversity

Unlike diffusion models that iteratively denoise a latent representation, autoregressive visual generation samples tokens sequentially in a raster-scan order. This provides a natural interface for joint text-image reasoning but introduces distinct trade-offs.

• Sequential sampling: Tokens are generated left-to-right, top-to-bottom, which can cause error accumulation in long sequences
• Inference speed: Typically slower than a single diffusion denoising step but benefits from standard LLM inference optimizations like KV-caching
• Hybrid approaches: Some models use autoregressive generation for the overall layout and diffusion for high-resolution refinement

The unified sequence model can perform few-shot learning across modalities. By providing interleaved text-image examples in the prompt, the model adapts to new tasks without fine-tuning. For instance, showing a few examples of a new visual style conditions the model to generate images in that style.

• Prompt structure: A sequence of demonstration pairs: [text_input_1, image_output_1, text_input_2, image_output_2, ...]
• Emergent capability: The model learns to map text instructions to visual outputs from context alone
• Limitation: Context window size restricts the number of examples, especially with high-resolution images consuming many tokens

The foundational visual encoder that makes unified generation possible. A Vision Transformer applies a pure transformer architecture directly to sequences of image patches, treating them as visual tokens. This patch-based approach allows images to be processed with the same self-attention mechanisms as text, enabling a single architecture to handle both modalities. ViTs are typically pre-trained on massive image datasets and serve as the visual backbone in multimodal autoregressive models.

The critical preprocessing step that converts a raw image into a sequence of tokens compatible with a transformer. The process involves:

• Dividing an image into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels).
• Flattening each patch into a 1D vector.
• Linearly projecting this vector into the model's embedding dimension.
• Adding a learned positional embedding to retain spatial information. This transforms a 2D visual signal into a 1D sequence indistinguishable in format from a sentence of text tokens, enabling a single autoregressive model to predict the next element, whether it's a word or a patch.

A shared high-dimensional vector space where representations of text tokens and image patches are projected to enable direct comparison and joint prediction. In a unified space, the vector for the word 'cat' is geometrically close to the vectors for image patches containing a cat. This alignment is achieved through joint training on interleaved text-image data, allowing the autoregressive model to seamlessly transition between generating a text description and generating the visual pixels that match it.

A neural operation that allows one modality to inform the generation of another. In a multimodal transformer, queries from the text stream can attend to keys and values from the image stream (and vice-versa). For example, when generating a caption, the word 'blue' can use cross-attention to focus on the specific image patches that are blue. This mechanism is the core computational glue that fuses information from different modalities, ensuring that generated text is visually grounded and generated images are semantically controlled.

A reasoning technique that interleaves textual rationale with direct visual evidence. Instead of generating a final answer immediately, the model outputs a step-by-step reasoning trace that can reference specific image regions. For instance, when asked 'Is the red ball to the left of the blue cube?', the model might generate: 'First, I see a red ball at coordinates [x1,y1]. Next, I locate a blue cube at [x2,y2]. Comparing x-coordinates, x1 < x2, so yes.' This explicit grounding in visual tokens dramatically reduces hallucination in complex visual question answering.

The ultimate capability of a multimodal autoregressive model: generating a coherent, mixed sequence of text and images. The model predicts the next token from a unified vocabulary that includes both text tokens and visual patch tokens. This enables outputs like a tutorial that generates a descriptive paragraph, followed by a diagram, followed by more instructions, all in a single autoregressive pass. This is a step beyond separate text-to-image or image-to-text models, representing a truly integrated generative process.