Vision Transformer (ViT)

A ViT splits an input image into a fixed sequence of non-overlapping patches, which are linearly embedded. This is the fundamental shift from pixel-level convolutions to treating an image as a sequence of tokens, analogous to words in NLP.

• Process: An image of size H×W×C is divided into N = (H×W) / P² patches, each of size P×P×C.
• Linear Projection: Each patch is flattened and projected into a D-dimensional embedding space via a trainable linear layer.
• Positional Encoding: A learnable 1D positional embedding is added to each patch token to retain spatial information, as the transformer itself is permutation-invariant.

Inspired by the BERT [CLS] token, a ViT prepends a learnable classification token to the sequence of patch embeddings. This token aggregates global information from all other patches through the self-attention mechanism.

• Function: The class token's final state, after processing through the transformer encoder, serves as the global image representation used for classification.
• Mechanism: As it propagates through the layers, it attends to all image patches, integrating contextual information across the entire image.
• Output: The final hidden state of this token is fed into a small Multi-Layer Perceptron (MLP) head to produce classification logits.

The core of ViT is a stack of identical Transformer Encoder layers, identical to those used in NLP models like BERT and GPT. Each layer consists of two main sub-layers:

• Multi-Head Self-Attention (MHSA): Computes weighted relationships between all patch tokens, allowing the model to integrate information globally, regardless of spatial distance.
• MLP Block: A position-wise feed-forward network applied to each token independently after attention.
• LayerNorm & Residual Connections: Applied before each sub-layer, following the Pre-Norm configuration, which stabilizes training of deep architectures.

Unlike Convolutional Neural Networks (CNNs), ViT has minimal built-in spatial inductive bias. CNNs assume locality and translation equivariance via their convolutional kernels. ViT makes no such assumptions; it must learn spatial relationships entirely from data.

• Implication: This makes ViT highly flexible but requires large-scale datasets (like JFT-300M or ImageNet-21k) for pre-training to learn these relationships effectively.
• Benefit: The model can develop a more global understanding of an image from the start, as self-attention has a global receptive field in every layer.

A Hybrid ViT uses a CNN backbone (like ResNet) as a feature extractor before the transformer. Instead of raw image patches, the input sequence is formed from feature maps of the CNN's intermediate layers.

• Process: A CNN processes the image, and its output feature map is divided into spatial patches, which are then flattened and embedded.
• Use Case: This variant can be beneficial when pre-training data is limited, as the CNN provides a useful initial spatial prior.
• Performance: On mid-sized datasets like ImageNet, hybrid models can sometimes outperform pure ViT models trained from scratch.

ViT's performance scales predictably with model size, dataset size, and compute, following trends observed in large language models.

• Model Variants: Common configurations include ViT-Base (12 layers, 768 hidden dim), ViT-Large (24 layers, 1024 hidden dim), and ViT-Huge (32 layers, 1280 hidden dim).
• Computational Cost: The self-attention mechanism has quadratic complexity O(N²) with respect to the number of patches, which can be a bottleneck for high-resolution images. Techniques like windowed attention (used in Swin Transformers) are later innovations to address this.
• Transfer Learning: After large-scale pre-training, ViT models demonstrate excellent performance when fine-tuned on downstream tasks with fewer labeled examples.

ViT's primary and most direct application is image classification, where it categorizes an entire image into a single label (e.g., identifying a dog breed or a medical condition). By treating an image as a sequence of patches, the model's global self-attention mechanism allows it to weigh the importance of all patches relative to each other when making a decision, often outperforming convolutional neural networks (CNNs) on large-scale datasets like ImageNet. This makes it highly effective for tasks requiring holistic scene understanding.

Adaptations like Detection Transformer (DETR) and its successors demonstrate ViT's power in object detection. Instead of relying on region proposal networks, these models treat detection as a direct set prediction problem. The transformer encoder processes image features, and the decoder uses learned object queries to attend to these features and predict bounding boxes and class labels in parallel. This architecture simplifies the detection pipeline and reduces the need for hand-crafted components like non-maximum suppression.

ViT architectures excel at dense prediction tasks like semantic segmentation (labeling each pixel with a class) and instance segmentation (identifying and segmenting individual objects). Models such as Segmenter and MaskFormer use a ViT backbone to generate high-resolution feature maps. The self-attention mechanism is crucial for capturing long-range contextual dependencies between pixels, which is essential for accurately delineating object boundaries and understanding scene composition, especially in complex environments like urban street scenes or medical imagery.

Extending the patch paradigm to the temporal dimension, Video Vision Transformers (ViViT, TimeSformer) apply self-attention across both space and time. An input video is divided into spatiotemporal patches. The model can then reason about actions by attending to relevant regions across multiple frames simultaneously. This approach is foundational for:

• Action Recognition: Classifying human activities in video clips.
• Temporal Localization: Detecting the start and end times of specific actions within a longer video.
• Video Captioning: Generating descriptive text for video content.

ViT serves as a powerful visual encoder in multi-modal architectures that combine vision with other modalities like language. In models like CLIP (Contrastive Language-Image Pre-training), a ViT encodes images into a shared embedding space with text encoded by a transformer. This enables zero-shot transfer—the model can classify images into novel categories based on natural language descriptions without task-specific training. This paradigm is critical for building systems that understand and generate content across vision and language, such as visual question answering and image captioning.

ViTs are increasingly deployed in medical imaging due to their ability to model long-range dependencies in high-resolution scans. Key applications include:

• Diagnostic Classification: Detecting diseases from X-rays, MRIs, and CT scans.
• Anomaly Detection: Identifying tumors or lesions by attending to subtle, globally distributed patterns.
• Segmentation: Precisely outlining anatomical structures or pathological regions in 3D volumetric data. The global context provided by self-attention allows the model to correlate distant anatomical features, which is often critical for accurate diagnosis, outperforming CNNs that have a more localized receptive field.

Multi-Head Self-Attention (MHSA) is the core computational mechanism of the Vision Transformer. It allows the model to weigh the importance of different image patches relative to each other. Instead of a single attention function, MHSA runs multiple attention operations in parallel (the 'heads'), each learning to focus on different types of relationships within the sequence of patches.

• Key Function: Enables the model to capture diverse dependencies, such as long-range spatial relationships between distant patches, which is a limitation of traditional convolutional filters.
• ViT Application: In ViT, after an image is split into a sequence of flattened patches, MHSA computes interactions between all patches, allowing a patch in the top-left corner to directly influence a patch in the bottom-right corner in a single layer.

A Convolutional Neural Network (CNN) is the dominant pre-ViT architecture for computer vision, built on layers of convolutional filters that slide across an image to detect local patterns like edges and textures. Hierarchical layers build up to complex features.

• Inductive Bias: CNNs have built-in translation equivariance and locality, meaning they assume important features can appear anywhere and are composed of local patterns. This reduces the amount of data needed to learn effectively.
• Contrast with ViT: ViT dispenses with these convolutional inductive biases. It has no inherent concept of locality or 2D structure; it must learn all spatial relationships from scratch via self-attention, which typically requires significantly more training data than CNNs.

Patch Embedding is the critical first step that adapts the transformer architecture, designed for sequences of tokens, to process 2D images. It involves splitting an input image into a grid of fixed-size, non-overlapping patches and linearly projecting each patch into a lower-dimensional vector.

• Process: For a 224x224 image and a patch size of 16x16, this results in 196 patches (224/16 = 14, 14x14=196). Each 16x16x3 patch (768 values) is flattened and projected via a trainable linear layer into a D-dimensional embedding vector.
• Position Embeddings: Since transformers are permutation-invariant, a learnable position embedding is added to each patch embedding to retain the spatial order of the original image. This provides the model with the necessary positional context.

A Hybrid Architecture is a variant of the Vision Transformer that uses a CNN backbone (like ResNet) to extract a 2D feature map from the input image, which is then split into patches and fed into the transformer encoder. This combines the strengths of both paradigms.

• Rationale: The CNN acts as a sophisticated patch creator, extracting higher-level, semantically rich features than raw pixel patches. This can improve performance, especially when training data is limited, as it provides the transformer with a more useful starting representation.
• Use Case: Often used in early ViT adaptations or in domains where convolutional feature extraction is already well-established, providing a practical bridge between classical and transformer-based vision models.

Self-Supervised Learning (SSL) is a training paradigm where a model learns representations by solving a 'pretext task' created automatically from unlabeled data. For ViTs, SSL is crucial due to their high data hunger.

• Pretext Tasks for ViT: Common methods include Masked Autoencoding (e.g., MAE), where random patches of an image are masked, and the model must reconstruct the missing pixels. Another is DINO, which uses a student-teacher framework with different augmentations to learn robust features.
• Impact: SSL allows ViTs to be pre-trained on massive, uncurated image datasets (e.g., ImageNet-1K, or larger web-scraped sets) without expensive human labels. This pre-training learns general visual representations that can be effectively fine-tuned on smaller, labeled downstream tasks.

The [CLS] token (classification token) is a special, learnable vector prepended to the sequence of patch embeddings at the input of the Vision Transformer. Its state at the output of the transformer encoder is used as the aggregate representation of the entire image for classification.

• Mechanism: As the sequence of embeddings (patches + CLS token) passes through the transformer layers, the CLS token aggregates information from all other patches via the self-attention mechanism.
• Output: The final hidden state corresponding to the CLS token is fed into a small Multilayer Perceptron (MLP) head to produce the final classification logits. This is analogous to global average pooling in CNNs but is a dynamic, learned aggregation.