Vision Transformer (ViT)

A Vision Transformer splits an input image into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is linearly projected into a patch embedding, a vector that serves as the token for the transformer. This process converts the 2D image into a 1D sequence of tokens, analogous to words in a sentence, which the transformer can process. A class token is prepended to this sequence, whose final state is used as the image representation for classification tasks.

Since the transformer's self-attention mechanism is inherently permutation-invariant, it lacks an inherent notion of spatial order. Positional encodings are added to the patch embeddings to inject information about the original 2D location of each patch. These can be learned during training or use fixed sinusoidal functions. This allows the model to understand that a patch from the top-left corner is spatially distinct from one in the bottom-right, preserving the structural information of the image.

The core of the ViT is the Multi-Head Self-Attention (MHSA) mechanism. It allows each patch embedding to attend to, and integrate information from, every other patch in the image. This enables the model to capture long-range dependencies and global context from the very first layer, a key advantage over convolutional networks which build receptive fields gradually. Multiple attention heads operate in parallel, each learning to focus on different types of relationships (e.g., texture, shape, color) within the image.

A ViT encoder is a stack of identical blocks. Each block typically consists of:

• LayerNorm: Applied before each major sub-layer for stable training.
• Multi-Head Self-Attention: Computes weighted interactions between all patches.
• Residual Connection: Adds the input of a sub-layer to its output, aiding gradient flow.
• MLP (Multi-Layer Perceptron): A simple feed-forward network applied to each token independently after another LayerNorm. This architecture allows for deep, stable training and the progressive refinement of visual representations.

Convolutional Neural Networks (CNNs) have a strong inductive bias for images: translation equivariance (a feature detector works anywhere) and locality (processing local neighborhoods). ViTs have a much weaker inductive bias—they treat patches as a set with no inherent spatial assumptions. This makes them data-hungry, requiring large-scale datasets (like JFT-300M or ImageNet-21k) for pre-training to learn visual concepts from scratch. However, this flexibility can lead to superior performance on large datasets by learning more generalizable and globally-aware representations.

ViTs are often used in hybrid models where a CNN backbone extracts initial feature maps, which are then treated as the input sequence for the transformer. This leverages CNN's efficient low-level feature extraction with the transformer's powerful global reasoning. Beyond classification, ViT's architecture is foundational for:

• Object Detection (DETR)
• Image Segmentation (Segmenter, MaskFormer)
• Multi-Modal Models (CLIP, which aligns ViT image features with text)
• Video Understanding (by treating frames as a temporal sequence of patches)

The self-attention mechanism is the core computational unit of the transformer architecture. It allows a model to weigh the importance of different elements (e.g., image patches or words) in a sequence relative to each other when computing representations.

• In a Vision Transformer, each image patch attends to all other patches, enabling the model to capture long-range dependencies and global context across the entire image.
• This is a fundamental shift from convolutional neural networks (CNNs), which process information through local, sliding filters.
• The mechanism is calculated using queries, keys, and values, producing a weighted sum of values where the weights are determined by the compatibility between queries and keys.

Patch embedding is the process of converting a 2D image into a sequence of 1D vectors suitable for transformer input. This is the first and critical adaptation step in a Vision Transformer.

• The input image is divided into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels).
• Each patch is flattened and linearly projected into a D-dimensional embedding vector.
• A learnable class token embedding is prepended to this sequence, whose final state serves as the global image representation for classification tasks.
• Position embeddings are added to the patch embeddings to retain spatial information, as the transformer itself is permutation-invariant.

A Convolutional Neural Network (CNN) is the dominant neural architecture for image processing that ViT challenged. CNNs process data using hierarchical layers of convolutional filters that exploit translation equivariance and local spatial correlations.

• Key Contrast with ViT: CNNs have inductive biases for images (locality, spatial hierarchies) built into their architecture. ViT lacks these biases and must learn all spatial relationships from data via self-attention.
• Hybrid Models: Architectures like ConvNeXt modernize CNNs, and some ViT variants (e.g., Swin Transformer) reintroduce convolutional-like hierarchical processing for efficiency.
• Historically, CNNs like ResNet and EfficientNet were the backbone of computer vision before the advent of large-scale transformer-based models.

Multi-head attention is an extension of the self-attention mechanism where the model performs multiple, parallel self-attention operations ("heads") over different learned linear projections of the input.

• Each attention head can learn to focus on different types of relationships or aspects of the data (e.g., different objects, textures, or spatial relationships in an image).
• The outputs from all heads are concatenated and linearly projected to form the final output.
• This parallelization allows the model to capture a richer, more diverse set of features from the input sequence simultaneously. In ViT, multi-head attention is applied across the sequence of patch embeddings.

The Swin Transformer is a hierarchical vision transformer that introduces shifted windowing to bring greater efficiency and modeling power to ViT architectures.

• It addresses ViT's computational complexity (which scales quadratically with the number of patches) by computing self-attention within local, non-overlapping windows.
• Shifted windows between consecutive layers allow cross-window connections, enabling the model to capture global context while maintaining linear computational complexity relative to image size.
• This hierarchical design with merged patches at deeper layers mimics the multi-scale feature maps of CNNs, making it highly effective for dense prediction tasks like object detection and semantic segmentation.

A Masked Autoencoder (MAE) is a self-supervised pre-training framework for Vision Transformers inspired by masked language modeling in BERT.

• During training, a large random subset (e.g., 75%) of the input image patches is masked (removed).
• The Vision Transformer encoder only processes the small subset of visible patches.
• A lightweight decoder then reconstructs the original image pixels from the encoded visible patches and learnable mask tokens.
• This approach forces the model to learn powerful, generalizable representations by predicting missing content, leading to ViT models that achieve state-of-the-art performance with less labeled data.