Patch Embedding

The core operation divides an input image H × W × C into a grid of N non-overlapping patches of fixed size P × P. Each patch is flattened into a 1D vector of length P²·C and linearly projected to a D-dimensional embedding space. This transforms the image from a 3D tensor into a sequence of N token vectors, where N = HW/P². A learnable [CLS] token is prepended to the sequence, and 1D positional embeddings are added to retain spatial information lost during flattening.

Each flattened patch vector is mapped to the model's latent dimension D via a trainable linear transformation—effectively a fully connected layer without bias. This projection can be implemented as a 2D convolution with kernel size and stride equal to P, outputting D channels. The convolution approach is computationally equivalent but more efficient, performing the patch extraction and embedding in a single operation. The resulting weight matrix learns to encode low-level features like edges, textures, and color gradients directly from raw pixel values.

Since self-attention is permutation-invariant, patch embeddings require explicit spatial information. Standard ViT adds learnable 1D positional embeddings to each patch token after projection. These embeddings are initialized randomly and trained end-to-end, allowing the model to discover optimal position representations. Alternative approaches include 2D sinusoidal encodings that encode row and column coordinates separately, or relative position biases added directly to attention scores. Without positional encoding, the model cannot distinguish between a patch in the top-left corner and one in the center.

Advanced architectures like Swin Transformer and PVT extend basic patch embedding with hierarchical multi-scale representations. Instead of a single fixed patch size, these models:

• Start with small 4×4 patches for fine-grained local features
• Use patch merging layers to concatenate neighboring patch embeddings
• Progressively reduce spatial resolution while increasing channel dimension This pyramid structure produces feature maps at multiple scales—/4, /8, /16, /32 of input resolution—making them compatible with dense prediction tasks like segmentation and detection that require high-resolution outputs.

While standard ViT uses non-overlapping patches, some variants employ overlapping patch embedding via strided convolutions where stride < kernel_size. This creates patches that share pixels with neighbors, providing:

• Smoother spatial transitions between adjacent tokens
• Implicit local context overlap that aids boundary feature extraction
• Better preservation of fine-grained spatial relationships The trade-off is increased computational cost—more total patches and redundant computation—but the improved spatial continuity often benefits tasks requiring precise localization.

Patch embedding is the visual analog of word embedding in NLP transformers. In text processing, words are mapped to dense vectors via a lookup table; in vision, image patches are mapped via linear projection. Both produce a sequence of D-dimensional vectors fed into identical transformer encoder blocks. This design unification is the key insight of Vision Transformers: treating images as a foreign language where each patch is a 'visual word.' The same self-attention mechanism then models relationships between patches just as it models relationships between words.

The foundational architecture that introduced patch embedding to computer vision. A ViT splits an image into fixed-size patches, linearly embeds each patch, adds positional embeddings, and feeds the resulting sequence into a standard Transformer encoder. This design treats image patches analogously to word tokens in NLP, enabling the application of self-attention mechanisms directly to visual data without convolutional inductive biases.

The mathematical operation at the core of patch embedding. Each flattened image patch is multiplied by a learnable weight matrix E to project it into a D-dimensional embedding space. For a patch of size P×P×C (height, width, channels), the projection maps from R^(P²·C) to R^D. This is equivalent to a 2D convolution with stride equal to patch size, making it efficiently implementable in standard deep learning frameworks.

A critical companion to patch embedding that injects spatial location information into the sequence. Because self-attention is permutation-invariant, the model has no inherent notion of patch order. Positional embeddings—typically learned 1D vectors added to each patch embedding—encode the original grid position of each patch, enabling the transformer to reconstruct spatial relationships and understand global image structure.

A special learnable embedding prepended to the sequence of patch embeddings, inspired by BERT's [CLS] token. This token aggregates global image-level information through the self-attention layers and serves as the image representation for final classification. The CLS token interacts with all patch embeddings at every transformer layer, effectively pooling information from the entire image without requiring explicit global average pooling.

A variant where patch embeddings are extracted from intermediate feature maps of a CNN rather than raw pixels. Instead of splitting the original image, a ResNet backbone generates a feature map, and patches are extracted from this learned representation. This combines the local feature extraction strengths of convolutions with the global reasoning capabilities of transformers, often improving data efficiency and performance on smaller datasets.

Patch embedding is the visual equivalent of text tokenization in NLP transformers. Just as sentences are split into word or subword tokens and mapped to embeddings, images are divided into patches and projected into vectors. This unification enables multimodal transformers to process images and text through identical architectural backbones, treating both modalities as sequences of embeddings that can interact via cross-attention mechanisms.