Swin Transformer

The core innovation of Swin Transformer. Self-attention is computed strictly within non-overlapping local windows of a fixed size (e.g., 7×7 patches). To enable cross-window connections, the window partition shifts by half the window size between consecutive blocks. This cyclic shift mechanism allows information to propagate across the entire image without the quadratic complexity of global self-attention, reducing computation from O(N²) to O(N) relative to image resolution.

Unlike the standard Vision Transformer which maintains a constant feature resolution, Swin builds a multi-scale feature hierarchy by progressively merging adjacent patch tokens in deeper layers. Starting from 4×4 pixel patches, the network reduces spatial resolution by 2× at each stage while doubling the channel dimension. This produces feature maps at 4×, 8×, 16×, and 32× downsampling ratios, making it a natural drop-in replacement for convolutional backbones in detection and segmentation frameworks like Mask R-CNN and UPerNet.

Between hierarchical stages, Swin applies a patch merging layer that concatenates the features of each group of 2×2 spatially adjacent patches and projects them through a linear layer. This operation reduces the number of tokens by a factor of 4 while doubling the feature dimension, analogous to spatial pooling in convolutional networks. The design preserves spatial structure during resolution reduction, unlike the token dropping or strided attention used in other architectures.

Swin incorporates a learnable relative position bias matrix added directly to the attention scores within each local window. For a window of size M×M, the bias parameterizes all possible relative positions along the x and y axes, resulting in a (2M-1)×(2M-1) bias matrix. This explicit positional encoding is more effective than absolute position embeddings for dense prediction tasks and is invariant to the absolute location of the window within the image.

The shifted window partitioning creates windows of varying sizes at the image boundaries. To maintain efficient batched computation, Swin uses a cyclic shift operation that moves boundary windows to the top-left of the feature map, creating a consistent set of M×M windows. A corresponding attention mask prevents cross-window attention between non-adjacent regions that were artificially brought together by the shift. After attention computation, a reverse cyclic shift restores the original spatial arrangement.

By confining self-attention to fixed-size local windows, Swin achieves linear computational complexity with respect to image resolution. For an image with h×w patches and a window size of M, the complexity per block is O(h·w·M²) rather than O((h·w)²). With M typically set to 7, this makes Swin practical for high-resolution inputs like 1536×1536 medical images where global self-attention would be computationally prohibitive. This efficiency is critical for 3D volumetric analysis and whole slide imaging.

The core innovation of the Swin Transformer that enables cross-window connections while maintaining linear computational complexity. Standard window-based self-attention lacks connections between windows, limiting modeling power. Swin introduces a shifted window partitioning scheme that alternates between regular and shifted window configurations in consecutive blocks.

• Window size: Typically 7×7 patches, computing self-attention only within each local window
• Shift amount: Windows are displaced by half the window size in the subsequent block
• Efficient batch computation: Cyclic shifting and masking enable batched computation of all windows
• Complexity: Linear O(N) with respect to image resolution, compared to O(N²) for global self-attention

Unlike the standard Vision Transformer which maintains a constant feature map resolution, Swin constructs a multi-scale feature hierarchy by progressively merging image patches in deeper layers. This design is critical for dense prediction tasks like segmentation and detection.

• Stage 1: 4× downsampling, H/4 × W/4 resolution
• Stage 2: 8× downsampling via patch merging, H/8 × W/8 resolution
• Stage 3: 16× downsampling, H/16 × W/16 resolution
• Stage 4: 32× downsampling, H/32 × W/32 resolution

This pyramid structure makes Swin a natural drop-in replacement for convolutional backbones in frameworks like Mask R-CNN and U-Net.

The downsampling operation between Swin stages that reduces spatial resolution while doubling channel dimension. Patch merging concatenates features from 2×2 groups of neighboring patches and applies a linear projection.

• Input: H × W × C feature map
• Operation: Concatenate features of each 2×2 patch group → 4C-dimensional vector
• Projection: Linear layer reduces to 2C dimensions
• Output: H/2 × W/2 × 2C feature map

This operation is analogous to pooling layers in CNNs but is learned end-to-end, allowing the network to adaptively determine how to combine spatial information.

Swin Transformer augments the self-attention computation with a learnable relative position bias added to the attention logits before softmax. This bias encodes the spatial relationship between every pair of tokens within a window.

• Bias matrix B: Shape (2M-1) × (2M-1) for a window of size M, parameterized by relative coordinate offsets
• Integration: Attention = Softmax(QKᵀ/√d + B)V
• Interpolation: The bias matrix can be bicubically interpolated for fine-tuning at different window sizes

This inductive bias significantly outperforms absolute positional encodings and requires no additional computation during inference, as the bias values are fixed after training.

An upgraded architecture addressing training instability when scaling Swin to 3 billion parameters on massive datasets. Three key modifications enable stable training of extremely large vision models.

• Residual post-norm: Moves LayerNorm from the start of each block to after the residual connection, producing much milder activation values
• Scaled cosine attention: Replaces dot-product attention with scaled cosine similarity: Sim(q,k) = cos(q,k)/τ, where τ is a learnable scalar
• Log-spaced continuous position bias: Replaces the discrete relative position bias with a continuous meta-network that takes log-spaced relative coordinates as input, enabling smooth transfer across window sizes

These techniques enabled training SwinV2-G with 3B parameters on the ImageNet-22K dataset.

A U-shaped architecture that combines a Swin Transformer encoder with a CNN-based decoder for volumetric medical image segmentation. Designed specifically for 3D modalities like CT and MRI.

• Encoder: Swin Transformer processes 3D tokens extracted from volumetric patches, computing self-attention within shifted windows across all three spatial dimensions
• Decoder: CNN-based with skip connections at multiple resolutions
• Multi-scale feature extraction: Leverages Swin's hierarchical design to capture features at five different resolutions
• Application: State-of-the-art performance on BTCV multi-organ segmentation and BraTS brain tumor segmentation benchmarks

The architecture effectively balances the global context modeling of transformers with the local precision of convolutional decoders.