Vision Transformer (ViT)

A ViT's most defining operation is splitting an input image into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is linearly projected into a patch embedding, a 1D vector. These embeddings, combined with a learnable [CLS] token and position embeddings, form the input sequence for the transformer. This converts the 2D spatial structure of an image into a 1D sequence the transformer can process, analogous to words in a sentence.

Inspired by the [CLS] token in BERT, ViT prepends a learnable classification token to the sequence of patch embeddings. This token interacts with all other patches via the transformer's self-attention mechanism. By the final layer, its state aggregates global information from the entire image. The output corresponding to this token is fed into a small MLP head to produce the final image classification, making it the model's holistic scene representation.

Since the transformer architecture is inherently permutation-invariant, position embeddings are added to the patch embeddings to retain spatial information. ViT uses standard 1D learnable embeddings, where each patch position gets a unique vector. While this discards explicit 2D relationships, the model learns relative spatial configurations through attention. Advanced variants explore 2D-aware or relative position embeddings for better inductive bias on images.

ViT demonstrates a key transformer property: performance scales predictably with model size (parameters) and training data. On large datasets (e.g., JFT-300M), large ViT models outperform state-of-the-art convolutional networks (CNNs) on tasks like ImageNet classification. This scalability is due to the transformer's efficient use of compute and its ability to model long-range dependencies across all patches simultaneously, unlike CNNs' local receptive fields.

The core of the ViT encoder is the multi-head self-attention (MSA) mechanism. For each patch, self-attention computes a weighted sum of information from all other patches. This allows any patch to directly influence any other, enabling the model to integrate global context from the start. For example, to identify a 'dog', the model can attend the dog's head, tail, and body patches regardless of their spatial separation, building a coherent object representation.

A practical variant replaces the raw image patch projection with feature maps from a CNN backbone (e.g., ResNet). The CNN acts as a feature extractor, and its output feature map is then split into patches for the transformer. This hybrid model leverages the CNN's innate 2D inductive bias and hierarchical feature learning for early processing, which can be beneficial when training data is limited, while still gaining the transformer's global reasoning benefits.

The original and most direct application of ViT. The model treats an image as a sequence of patches, classifies the entire image into a single category (e.g., 'golden retriever', 'aircraft carrier'). Key advantages include:

• Global context modeling: Unlike CNNs which have limited receptive fields in early layers, ViT's self-attention mechanism allows every patch to attend to every other patch from the first layer, capturing long-range dependencies crucial for scene understanding.
• Scalability with data and compute: ViT performance scales predictably with larger model sizes and datasets, often outperforming CNNs when trained on massive datasets like JFT-300M.
• Standard benchmarks: Dominates leaderboards on ImageNet, CIFAR-100, and other classification datasets.

ViT serves as a powerful backbone within detection transformer (DETR) architectures for end-to-end object detection.

• Set prediction: Replaces traditional region proposal networks (RPN) and non-maximum suppression (NMS) with a transformer decoder that directly predicts a set of object boxes and classes.
• Encoder backbone: A ViT encoder processes the image into a global feature representation. A transformer decoder then takes learned object queries and attends to these features to produce final detections.
• Panoptic segmentation: Extended frameworks like Mask DETR use ViT backbones to perform unified instance and semantic segmentation, predicting masks for each detected object.

ViT architectures are extended to process spatiotemporal data by treating video as a sequence of spatial-temporal tokens.

• Factorized encoders: Models like TimeSformer separate spatial and temporal attention to efficiently process long video clips. Spatial attention is applied within each frame, and temporal attention is applied across frames for the same spatial location.
• Action recognition: Classifies human activities (e.g., 'playing violin', 'running') by modeling interactions between objects and their motion over time.
• Video object segmentation: Tracks and segments objects across frames by propagating patch-level features through time using attention mechanisms.

ViTs are increasingly applied to radiology and pathology due to their ability to capture global context in high-resolution images.

• Whole-slide image analysis: In digital pathology, a single slide can be 100,000x100,000 pixels. ViTs can process gigapixel images by hierarchically attending to patches at multiple resolutions, identifying disease patterns across large tissue areas.
• 3D medical imaging: For CT and MRI scans, 3D ViTs treat volumetric patches as tokens, enabling the model to understand relationships across different anatomical planes (axial, coronal, sagittal).
• Applications: Tumor detection and segmentation, disease classification (e.g., diabetic retinopathy grading), and biomarker prediction.

ViT is the standard visual encoder in state-of-the-art multimodal large language models (MLLMs).

• Visual feature extraction: A ViT encodes an image into a sequence of patch embeddings. These are projected into the same latent space as text tokens from a language model (e.g., LLaMA, GPT).
• Cross-modal alignment: Models like LLaVA and Florence-2 use a ViT to create visual tokens that a large language model can attend to, enabling tasks like visual question answering, image captioning, and visual dialogue.
• Contrastive pre-training: Foundational models like CLIP use a ViT image encoder paired with a text encoder, trained on 400M+ image-text pairs with a contrastive loss to align visual and linguistic concepts.

ViTs excel at analyzing satellite and aerial imagery where understanding large-scale spatial patterns is critical.

• Land cover classification: Categorizing every pixel in a satellite image into classes like 'forest', 'urban', 'water', or 'agriculture' by modeling long-range dependencies between different land regions.
• Change detection: Identifying differences between two satellite images of the same location taken at different times (e.g., deforestation, urban expansion) by comparing patch-level representations.
• Disaster response: Rapid assessment of flood, fire, or earthquake damage by analyzing pre- and post-event imagery. The global attention mechanism helps contextualize localized damage within the broader scene.

A Multimodal Large Language Model is a foundation model that extends the capabilities of a large language model to understand and generate content across multiple modalities, such as text and images. Unlike a pure ViT, an MLLM uses a ViT or similar encoder to process visual inputs into a sequence of tokens, which are then fed into a large language model's decoder for cross-modal reasoning and generation.

• Core Function: Acts as a unified interface for vision-and-language tasks like Visual Question Answering (VQA) and image captioning.
• Architecture: Typically employs a vision encoder (like ViT) and a language decoder (like a Transformer decoder), connected via a projection layer.
• Example: Models like LLaVA and Flamingo use this paradigm, where the ViT's output patches are treated as a prefix to the text token sequence.

Visual grounding is the computer vision task of linking linguistic concepts, such as words or phrases, to specific regions or objects within an image or video. It is the foundational capability required for models to understand referring expressions like 'the red car on the left'.

• Key Tasks: Includes Referring Expression Comprehension (REC) and Phrase Grounding.
• ViT's Role: Modern approaches often use a ViT backbone to extract dense visual features, which are then aligned with text embeddings from a language model via cross-attention mechanisms.
• Evaluation: Measured by how accurately a model can draw a bounding box or segmentation mask around the region described by the text.

Visual Question Answering is a multimodal task where a model must answer a natural language question based on the content of an input image. It requires joint understanding of vision and language, as well as often some degree of commonsense reasoning.

• Challenge: Questions can range from simple ('What color is the car?') to complex ('Why is the person holding an umbrella?').
• Modern Approach: State-of-the-art VQA systems are built on Multimodal LLMs that use a ViT-based vision encoder to convert the image into tokens consumable by a language model.
• Benchmarks: Popular datasets include VQAv2, GQA, and VizWiz, which test robustness and real-world applicability.