Vision Transformer (ViT)

The input image is divided into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is flattened into a 1D vector and linearly projected to a constant latent vector size, creating a sequence of 'visual tokens' analogous to word embeddings in NLP. A learnable [CLS] token is prepended to this sequence, and 1D positional embeddings are added to retain spatial information.

The sequence of patch embeddings is processed by a standard Transformer encoder, identical to the architecture proposed in 'Attention Is All You Need'. This stack consists of alternating layers of:

• Multi-Head Self-Attention (MSA): Allows every patch to attend to every other patch globally from the very first layer.
• MLP Block: A two-layer feed-forward network with GELU activation. Layer Normalization is applied before each block (pre-norm), and residual connections are used throughout.

Unlike CNNs that build receptive fields locally and hierarchically, ViT employs global self-attention from the outset. This allows the model to immediately model long-range dependencies between distant image regions. The attention mechanism computes weighted sums of values based on the similarity between queries and keys, enabling the model to integrate information across the entire image in a single layer.

A learnable [CLS] token embedding is prepended to the input sequence. The final hidden state corresponding to this token serves as the aggregate image representation. This state is passed through a simple MLP classification head to produce the final class prediction. This design mirrors BERT's approach, where the [CLS] token acts as a summary of the entire input sequence.

The input sequence can be formed from feature maps produced by a standard CNN instead of raw image patches. In this hybrid model, the CNN backbone extracts spatial features, and the resulting feature map is flattened into a sequence for the Transformer. This combines the local feature extraction strength of convolutions with the global reasoning of self-attention.

ViT has significantly less image-specific inductive bias than CNNs. Only the patch extraction and positional embeddings inject locality and spatial structure. Consequently, ViT does not inherently understand translation equivariance or locality and must learn these from data. This makes ViT highly data-hungry, requiring pre-training on massive datasets like JFT-300M to outperform CNNs, but it scales more effectively with model and data size.

The foundational input processing step for a Vision Transformer. An input image is divided into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is then flattened and linearly projected into a D-dimensional vector. This sequence of patch embeddings, combined with learned positional encodings, serves as the input token sequence to the standard Transformer encoder, directly analogous to word tokens in NLP.

Unlike Convolutional Neural Networks (CNNs) which have a local receptive field, the Vision Transformer's core mechanism is multi-head self-attention. This operation allows every image patch to directly interact with every other patch in the very first layer. This enables the model to immediately model long-range dependencies and global context, capturing relationships between distant objects without requiring deep hierarchical feature extraction.

Because the self-attention mechanism is permutation-invariant, a Vision Transformer has no inherent sense of spatial order. To preserve the 2D structure of the image, learned positional embeddings are added to the patch embeddings. These vectors encode the spatial location of each patch, allowing the model to understand concepts like 'above,' 'below,' and 'adjacent to' without any convolutional inductive bias.

A key architectural distinction is the inductive bias. CNNs are built with strong priors for locality and translation equivariance. Vision Transformers, based on the standard Transformer architecture, have a much weaker image-specific inductive bias. They learn spatial relationships purely from data. This makes ViTs more data-hungry but allows them to scale exceptionally well, often outperforming CNNs when pre-trained on massive datasets like JFT-300M.

Vision Transformers serve as the visual backbone for advanced Vision-Language Models (VLMs). Their token-based output makes them a natural fit for fusion with text tokens. In architectures using cross-attention, a language model's text queries can attend directly to the ViT's patch keys and values, enabling fine-grained visual grounding for tasks like Visual Question Answering (VQA) and image captioning.

Vision Transformers exhibit excellent scaling behavior. Their performance does not saturate as quickly as CNNs when increasing model size and dataset volume. Standard practice involves large-scale pre-training on a classification task (e.g., ImageNet-21k) followed by fine-tuning on downstream tasks. Advanced variants like Data-efficient Image Transformers (DeiT) use knowledge distillation from a CNN teacher to achieve competitive results with less data.