Positional Encoding

The foundational method where a unique vector is assigned to each spatial location and added to the input embeddings. In the original Vision Transformer (ViT) , standard learnable 1D position embeddings are used.

• Mechanism: A learnable parameter matrix of shape [num_patches + 1, embedding_dim] is added to the patch tokens.
• Limitation: Fixed sequence length; cannot easily extrapolate to higher-resolution images at inference time without interpolation.
• Origin: Directly adapted from the sinusoidal and learned embeddings of the NLP Transformer.

Encodes the spatial relationship between pairs of tokens directly within the self-attention computation, rather than adding absolute coordinates to the input.

• Implementation: A learnable bias matrix B is added to the attention logits before the softmax: Attention = softmax(QK^T / √d + B).
• Key Advantage: Naturally handles variable image resolutions and is central to the Swin Transformer, where biases are computed within local windows.
• Parameterization: Bias values are indexed by the relative coordinate offsets (Δx, Δy) between the query and key patches.

Encodes position by rotating the query and key vectors in the embedding space. The dot product between them naturally decays with relative distance.

• Formulation: Applies a rotation matrix to vectors at position m: f(q, m) = q * e^(imθ). The attention score f(q, m) * f(k, n) depends only on the relative offset m - n.
• Benefit: Provides theoretical elegance by injecting position information without adding to the token embeddings, preserving vector norm.
• Usage: Widely adopted in large language models and increasingly in vision models as an alternative to learned absolute biases.

Generates position embeddings dynamically from the local neighborhood of a patch using a small convolutional network, rather than using static learned parameters.

• Mechanism: A Depth-wise Convolution or a 3x3 Conv2D layer is applied to the reshaped feature map to produce position-aware embeddings.
• Zero-Shot Resolution Change: CPEs can seamlessly generalize to images of arbitrary sizes not seen during training because the generator is translation-equivariant.
• Architecture: Used in CPVT and PVTv2 to replace absolute position embeddings entirely, merging the benefits of CNNs and Transformers.

Extends spatial encoding to the depth dimension for CT and MRI scans. Patches are extracted from 3D volumes, requiring a coordinate tuple (x, y, z).

• Decomposition Strategy: To reduce parameters, 3D positions are often factorized into separate 1D encodings for height, width, and depth, which are summed together.
• Medical Context: Essential for models like UNETR and Swin UNETR, where understanding the spatial continuity of organs across axial slices is critical for accurate segmentation.
• Complexity: A 3D volume of 96x96x96 patches requires 884,736 positional tokens, making learnable embeddings parameter-heavy and favoring relative or decomposed methods.

A deterministic, non-learnable encoding using sine and cosine functions of varying frequencies. The original Transformer used this for 1D sequences.

• Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d)).
• Property: Allows the model to easily attend to relative positions because PE(pos+k) can be represented as a linear function of PE(pos).
• Vision Adaptation: Less common in modern vision transformers than learnable or relative biases, but provides a useful parameter-free baseline for 2D grid extensions.