Vision-Language Model (VLM)

The foundational design pattern where separate modality encoders process images and text independently before projecting them into a unified embedding space. A Vision Transformer (ViT) encodes image patches while a text transformer processes tokens. The outputs are aligned via a contrastive loss, enabling direct similarity comparison. This architecture powers models like CLIP and enables efficient zero-shot classification and cross-modal retrieval without requiring cross-attention between modalities.

Advanced VLMs employ cross-attention layers where textual queries attend to visual keys and values, enabling deep information exchange between modalities. Key fusion strategies include:

• Early Fusion: Raw features combined at input layer, preserving fine-grained detail
• Late Fusion: Independent encoding with combination only before output, offering modularity
• Intermediate Fusion: Cross-attention at multiple transformer layers, balancing depth and efficiency This mechanism is critical for tasks like Visual Question Answering (VQA) where precise alignment between words and image regions is required.

Images are converted into model-digestible tokens through patch embedding: the input image is divided into fixed-size patches (e.g., 16x16 pixels), each linearly projected into a vector. These visual tokens are concatenated with text tokens to form a unified sequence. A learned [CLS] token or special separator tokens demarcate modality boundaries. This tokenization strategy allows a single transformer backbone to process interleaved image-text sequences autoregressively, enabling unified generation of both modalities.

VLMs are trained using contrastive learning on massive datasets of image-text pairs. The objective maximizes cosine similarity between matched pairs while minimizing it for mismatched pairs within a batch. This InfoNCE loss forces the model to learn semantically meaningful joint representations. The resulting embedding space exhibits remarkable properties: text queries retrieve relevant images, and images can be classified by comparing them to textual class descriptions without task-specific fine-tuning.

Next-generation VLMs treat visual and textual data as a unified token vocabulary, enabling a single autoregressive transformer to generate interleaved image-text sequences. The model predicts the next token—whether a text subword or an image patch—conditioned on all preceding tokens. This architecture supports tasks like:

• Image captioning: Generate text conditioned on visual tokens
• Grounded image generation: Produce images from text descriptions
• Interleaved document creation: Generate documents with embedded figures and diagrams

A critical training technique where input from a random modality is intentionally removed during pre-training. By forcing the model to perform tasks with incomplete information—such as answering questions without visual input or generating captions without text context—the model learns robust, non-spurious correlations. This prevents over-reliance on a single modality and improves performance on unimodal tasks while maintaining strong cross-modal capabilities. The dropout rate is typically scheduled to increase throughout training.

A foundational training paradigm that learns a joint embedding space by maximizing the cosine similarity between matched image-text pairs and minimizing it for mismatched pairs. CLIP uses a dual-encoder architecture with separate Vision Transformer (ViT) and text transformer towers. This enables zero-shot classification by comparing an image's embedding to text embeddings of class labels, bypassing the need for task-specific fine-tuning.

A neural operation that enables information flow between modalities. Queries from one modality (e.g., text tokens) attend to keys and values from another (e.g., image patch embeddings). This is the core fusion technique in models like Flamingo and LLaVA, allowing language models to dynamically extract relevant visual features for each generated token rather than relying on a single static image representation.

Fine-tuning a pre-trained VLM on a dataset of task instructions paired with multimodal inputs and outputs. This bridges the gap between next-token prediction and helpful assistant behavior. Datasets like LLaVA-Instruct use language-only GPT-4 to generate diverse question-answer pairs about images, teaching the VLM to follow complex visual reasoning commands, perform Visual Question Answering (VQA) , and generate structured outputs.

The task of localizing the specific image region that corresponds to a natural language expression. Unlike object detection, which uses fixed class labels, visual grounding maps free-form text to bounding boxes or segmentation masks. This capability is critical for embodied agents and is often evaluated on benchmarks like RefCOCO. Modern VLMs achieve this by generating coordinate tokens directly in their text output.

An architecture that grounds language model generation in retrieved multimodal data. A query is embedded and used to search a unified embedding space containing both text chunks and images. Retrieved results are fed into the VLM's context window to provide factual grounding. This is essential for answering questions about proprietary documents containing charts, diagrams, and photographs where text-only retrieval would fail.

The standard visual encoder for modern VLMs. An image is divided into fixed-size patches (e.g., 16x16 pixels), which are linearly projected into a sequence of vectors. These patch embeddings, combined with positional encodings, are processed by a pure transformer architecture. This treats an image as a sequence of tokens, analogous to text, enabling seamless integration with language model backbones.