Vision Transformer (ViT)

A ViT splits an input image into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is flattened into a 1D vector and linearly projected into a model dimension. This transforms the 2D image into a 1D sequence of patch embeddings, analogous to word tokens in NLP. This fundamental step allows the standard transformer encoder, which expects sequential data, to process visual information.

• Example: A 224x224 image divided into 16x16 patches yields 196 sequence tokens.
• Key Benefit: It replaces the inductive bias of local connectivity in CNNs with a more flexible, global processing mechanism from the first layer.

Since the transformer's self-attention mechanism is inherently permutation-invariant, positional encodings are added to the patch embeddings to retain spatial information about where each patch originated in the original image. These are typically learnable 1D vectors, one per patch position. Without this, the model would have no notion of the original 2D structure, treating a patch from the top-left the same as one from the bottom-right.

• Contrast with CNNs: CNNs inherently capture spatial relationships through convolutional filters; ViTs must explicitly learn this through positional embeddings.
• Result: The model learns to attend to patches based on both content and their learned spatial relationships.

Inspired by the [CLS] token in BERT, a ViT prepends a special learnable embedding to the sequence of patch tokens. This class token aggregates global information from the entire image through the transformer's self-attention layers. After processing, the final state of this class token is used as the image representation for classification tasks, fed into a small MLP head.

• Function: Acts as a summary vector that attends to all relevant patches.
• Architectural Role: Provides a single, fixed-dimension vector for the whole image, analogous to the pooled output of a CNN's final feature map.

The core of a ViT is a stack of identical transformer encoder layers. Each layer consists of:

• Multi-Head Self-Attention (MSA): Allows each patch to attend to every other patch, capturing long-range dependencies across the entire image from the first layer.
• LayerNorm and Residual Connections: Applied before (pre-norm) or after each sub-layer for stable training.
• MLP Block: A position-wise feed-forward network applied to each token independently.

This architecture forgoes convolutional layers entirely, relying on self-attention for feature integration. Its global receptive field is a key differentiator from the local, hierarchical receptive fields of CNNs.

A Hybrid ViT uses a CNN backbone (e.g., ResNet) as a feature extractor instead of the simple patch projection. The CNN processes the image into a feature map, which is then flattened into a sequence of feature patches for the transformer encoder. This leverages the powerful, spatially hierarchical representations learned by CNNs before applying the transformer's global attention.

• Use Case: Can be beneficial when applying ViT to smaller datasets, as it incorporates proven CNN inductive biases.
• Trade-off: Introduces the complexity of a two-stage architecture but can improve data efficiency.

A defining characteristic of the original ViT is its data-hungry nature. When trained on mid-sized datasets like ImageNet, ViTs often underperformed comparable CNNs. However, when pre-trained on very large datasets (e.g., JFT-300M with 300M images), ViTs outperformed state-of-the-art CNNs while requiring substantially less computational pre-training. This demonstrates favorable scaling laws: performance improves predictably with increased model size and data.

• Key Insight: ViTs have fewer image-specific inductive biases than CNNs, so they require more data to learn generalized visual representations.
• Implication: ViTs established that the transformer's scalability, proven in NLP, directly translates to vision at sufficient scale.

The ViT's fundamental innovation is treating an image as a sequence of flattened patches. A standard 224x224 image is split into 196 non-overlapping 16x16 patches. Each patch is linearly projected into a patch embedding, combined with a learnable positional embedding to retain spatial information, and prepended with a class token. This sequence is then processed by a stack of standard Transformer encoder layers, which apply multi-head self-attention and feed-forward networks. The final representation from the class token is used for classification via an MLP head. This design foregoes convolutional inductive biases entirely, relying on the transformer's global attention to learn spatial relationships from data.

Several influential variants have refined the original ViT design:

• DeiT (Data-efficient Image Transformer): Introduced a distillation token and leveraged a teacher CNN to achieve strong performance with less data.
• Swin Transformer: Introduced a hierarchical architecture with shifted windows, enabling efficient computation at multiple scales and making it suitable for dense prediction tasks like object detection and segmentation.
• MAE (Masked Autoencoder): A self-supervised pre-training approach where a high proportion (e.g., 75%) of image patches are masked. The encoder processes only visible patches, and a lightweight decoder reconstructs the original image, learning powerful representations.
• ViT-G/14: A giant-scale variant demonstrating that scaling laws apply to vision transformers, achieving state-of-the-art accuracy with 2 billion parameters. These variants address core challenges in data efficiency, computational complexity, and representation learning.

ViTs are the backbone for modern image classification on benchmarks like ImageNet. Their ability to model long-range dependencies allows them to outperform traditional CNNs, especially on large datasets. In production, they power:

• Content moderation systems that classify images for safety.
• Medical image analysis for identifying pathologies in X-rays and histology slides.
• Industrial quality inspection by detecting defects in manufactured goods.
• Retail product categorization from user-uploaded photos. The model's classification head can be easily swapped for fine-tuning on domain-specific datasets, leveraging its powerful pre-trained visual features.

The ViT encoder is the standard visual backbone for nearly all state-of-the-art multimodal systems. It provides the dense, semantic visual features that are aligned with text. Key examples include:

• CLIP: Uses a ViT (or a CNN) as its image encoder, trained with a contrastive loss to align images and text in a joint embedding space.
• BLIP-2: Employs a frozen ViT (like ViT-G) and a frozen LLM, connecting them with a lightweight, trainable Querying Transformer (Q-Former) to enable efficient vision-language instruction following.
• Flamingo: Integrates a pretrained, frozen ViT with a large language model using perceiver resampler layers to condition the LLM on visual inputs. In these architectures, the ViT acts as a powerful, general-purpose visual feature extractor.

While the original ViT was designed for classification, its architecture has been adapted for dense prediction tasks that require per-pixel outputs, such as semantic segmentation and object detection.

• Segment Anything Model (SAM): Uses a ViT-based image encoder to produce an image embedding, which is combined with prompt encodings to generate high-quality object masks in a promptable, zero-shot manner.
• DETR and its variants: Replace traditional CNN backbones with a ViT to provide image features to a transformer-based detection head, formulating object detection as a set prediction problem.
• U-Net-like architectures: Use a ViT encoder paired with a convolutional or transformer-based decoder to upsample features for precise segmentation maps. These applications demonstrate the ViT's versatility as a foundational visual feature extractor beyond simple classification.

The sequential processing nature of transformers makes ViTs naturally extensible to spatiotemporal data.

• Video Classification: A common approach is uniform frame sampling, where a ViT independently encodes frames and a temporal transformer models interactions across time. Alternatively, 3D patches (tubelets) can be extracted from the video volume.
• Action Recognition: Models like TimeSformer factorize spatial and temporal attention within the transformer blocks to efficiently process video clips.
• 3D Vision: For tasks like point cloud classification, points can be voxelized or grouped into tokens, and a ViT-like architecture processes them. Multiview transformers also render 3D objects from multiple 2D views and process them with a ViT. These extensions highlight the transformer's flexibility in handling data beyond 2D images.

A Convolutional Neural Network (CNN) is a class of deep neural networks most commonly applied to analyzing visual imagery. It uses convolutional filters that slide across the input to extract hierarchical spatial features (edges, textures, objects).

• Architectural Contrast: CNNs use inductive biases like translation equivariance and local connectivity, which ViTs explicitly forgo.
• Historical Context: CNNs (e.g., ResNet, VGG) were the dominant architecture for computer vision until the advent of Vision Transformers.
• Hybrid Models: Many early ViT variants used a CNN backbone as a "patch extractor" before the transformer encoder.

Self-attention is the core computational mechanism within a transformer layer that allows a model to weigh the importance of all other elements in a sequence when encoding a particular element.

• Global Context: Unlike CNNs' local receptive fields, self-attention gives each image patch a direct, weighted connection to every other patch, enabling modeling of long-range dependencies.
• Scaled Dot-Product: The operation computes a weighted sum of values, where weights are derived from the compatibility of a query with a set of keys.
• Multi-Head Attention: ViTs use multiple parallel self-attention "heads" to jointly attend to information from different representation subspaces.

Masked Image Modeling (MIM) is a self-supervised pre-training objective for vision models where a random subset of input image patches is masked, and the model is trained to reconstruct the missing content.

• ViT Pre-training: While the original ViT paper used supervised pre-training on ImageNet, MIM (e.g., in BEiT, MAE) became a dominant, more scalable self-supervised approach for ViTs.
• Analogy to NLP: It is the visual counterpart to Masked Language Modeling (MLM) used in BERT.
• Objective: The model learns rich, contextual visual representations by predicting the properties (e.g., pixel values, tokenized visual tokens) of the masked patches.

The Swin Transformer is a hierarchical vision transformer that introduces shifted windows to compute self-attention, bringing greater efficiency and the ability to model at multiple scales.

• Hierarchical Feature Maps: Unlike ViT's single-scale sequence, Swin produces feature maps at multiple resolutions (like CNNs), making it suitable for dense prediction tasks like object detection and segmentation.
• Windowed Attention: Self-attention is computed within non-overlapping local windows, drastically reducing computational complexity from quadratic to linear relative to image size.
• Shifted Windows: A shifting mechanism between layers creates cross-window connections, restoring the global modeling capability.

Multi-Head Self-Attention (MHSA) is the specific self-attention module used in transformers, where the input is projected into multiple sets of queries, keys, and values, allowing the model to attend to information from different subspaces simultaneously.

• ViT Implementation: A standard ViT block uses MHSA. For an input sequence of patch embeddings, it computes attention across all patches in multiple parallel heads.
• Benefits: Different heads can learn to focus on different types of spatial relationships (e.g., texture, shape, long-range object connections).
• Formula: MHSA(Q,K,V) = Concat(head₁, ..., headₕ)Wᴼ where headᵢ = Attention(QWᵢᴼ, KWᵢᴷ, VWᵢⱽ).

DeiT is a variant of the Vision Transformer designed to be trained efficiently without requiring massive, private datasets. It introduced a teacher-student distillation strategy specifically for transformers.

• Key Innovation: Uses a distillation token that interacts with patch tokens and class token in the transformer, learning to mimic the output of a powerful CNN teacher (e.g., RegNet).
• Training Efficiency: Achieved competitive ImageNet accuracy using only the ImageNet-1k dataset, without the large JFT-300M dataset required by the original ViT.
• Impact: Demonstrated that ViTs could be practical for research and applications without exascale computational resources.