Patch Embedding

An input image x ∈ R^(H×W×C) is reshaped into a sequence of flattened 2D patches x_p ∈ R^(N×(P²·C)), where (P, P) is the patch resolution and N = HW/P² is the resulting number of tokens. Each patch is then mapped to a latent D-dimensional embedding via a trainable linear projection E ∈ R^((P²·C)×D). This is mathematically equivalent to a 2D convolution with a stride and kernel size equal to the patch size, making it a fully learned, non-overlapping feature extractor.

Unlike CNNs which inherently capture spatial structure via sliding filters, the subsequent transformer self-attention is permutation-invariant. To retain spatial inductive bias, learnable 1D position embeddings are added to the patch embeddings. This allows the model to distinguish patch_2,3 (row 2, column 3) from patch_3,2. Standard ViTs use absolute position embeddings, while later variants explore relative positional biases or 2D-aware encodings to improve resolution generalization.

Following BERT's design, a learnable [class] token embedding is prepended to the sequence of patch embeddings before entering the transformer encoder. The final hidden state corresponding to this token serves as the aggregate image representation y for classification heads. This avoids imposing a specific pooling structure on the patch-level outputs and allows the transformer to dynamically weigh information from all patches through self-attention.

The number of patches N is a quadratic function of input resolution: N = (H/P) × (W/P). This creates a direct compute-performance trade-off. A 224×224 image with P=16 yields 196 tokens, while a 384×384 image yields 576 tokens. Since self-attention complexity is O(N²), higher resolutions dramatically increase memory usage. Techniques like 2D interpolation of pre-trained position embeddings are required to fine-tune models on resolutions unseen during pre-training.

Instead of raw pixel patches, a standard ResNet feature map can serve as the input sequence. In this hybrid architecture, the patches are formed from a lower-resolution convolutional feature map (e.g., stride-32 output), where each spatial position is treated as a 'patch' embedding. This injects stronger local inductive biases like translation equivariance directly into the token stream, often improving sample efficiency and accuracy on smaller datasets like ImageNet-1k.

The patch size P defines the granularity of the initial visual vocabulary. A small patch size (e.g., P=8) preserves fine-grained textures and small objects but generates a long sequence (e.g., 784 tokens), increasing compute. A large patch size (e.g., P=32) creates a coarse, highly compressed token sequence (e.g., 49 tokens), which is computationally cheap but risks losing small-scale structural details crucial for dense prediction tasks like semantic segmentation.

The foundational architecture that applies a pure transformer encoder directly to sequences of image patches. Unlike CNNs, ViT treats an image as a sequence of visual tokens, where each token is the linear projection of a flattened patch. This design removes the inductive bias of convolution, allowing the model to learn global spatial relationships through self-attention from the very first layer. ViT achieves state-of-the-art results on image classification when pre-trained on large datasets like JFT-300M or ImageNet-21k.

A critical companion to patch embedding that injects spatial location information into the sequence of patch tokens. Since self-attention is permutation-invariant, the transformer has no inherent notion of which patch belongs where. Positional embeddings—typically learned 1D vectors added to each patch token—encode the row-column coordinates of each patch in the original image grid. Without them, the model would treat the image as a bag of unordered patches, losing all spatial structure.

The mathematical engine behind patch embedding. For an image patch of size P × P × C (height, width, channels), this layer flattens the patch into a 1D vector of length P²C and multiplies it by a learned weight matrix W ∈ ℝ^(P²C × D) to produce a D-dimensional embedding. This is equivalent to a 2D convolution with kernel size P and stride P, followed by a flattening operation. The projection maps raw pixel intensities into a dense, semantically meaningful latent space.

A learnable embedding prepended to the sequence of patch tokens, following BERT's design. This special token aggregates global image information through self-attention across all layers. After the final transformer block, the [CLS] token's representation serves as the image-level feature vector for classification tasks. It has no associated patch location and is initialized randomly, learning to pool relevant information from all patches during training.

An architecture that extends patch embedding to fuse visual and textual modalities. Image patches are embedded as visual tokens, while text is tokenized into word embeddings. These two token sequences are concatenated and processed by a unified transformer using cross-attention mechanisms. This enables tasks like visual question answering and image captioning, where the model must reason jointly over pixel-level and semantic-level representations.

An evolution of fixed patch embedding used in architectures like Swin Transformer. Instead of maintaining a constant patch resolution, hierarchical models progressively merge adjacent patch tokens in deeper layers—reducing sequence length while increasing channel dimension. This creates a multi-scale feature pyramid analogous to CNN backbones, enabling efficient processing of high-resolution images and strong performance on dense prediction tasks like object detection and semantic segmentation.