Swin UNETR

The encoder is built upon the Swin Transformer architecture, which processes 3D volumetric data in a hierarchical fashion. It begins with a patch partition layer that creates non-overlapping tokens. The network consists of four stages, each containing a pair of Swin Transformer blocks. A patch merging layer between stages reduces the spatial resolution by a factor of 2 while doubling the feature dimension, creating a multi-scale feature pyramid analogous to a convolutional backbone.

• Linear complexity: Achieved by computing self-attention within non-overlapping local windows.
• Cross-window connection: The shifted window mechanism in successive blocks enables information propagation between windows.

The decoder is a purely convolutional design that reconstructs the segmentation map from the encoded feature hierarchy. At each resolution stage, the decoder block receives a feature map from the encoder via a skip connection. This feature map is first resized to match the decoder's spatial dimensions using transposed convolution, then concatenated with the decoder's upsampled features. A residual block with two convolutional layers processes the combined representation.

• Feature fusion: Skip connections inject fine-grained spatial detail lost during encoding.
• Final output: A 1x1x1 convolution followed by a softmax activation produces voxel-level class probabilities.

Swin UNETR is designed to leverage large-scale unlabeled datasets through self-supervised pre-training before fine-tuning on smaller annotated medical datasets. The pre-training tasks teach the encoder to learn meaningful anatomical representations without manual labels.

• Masked Volume Inpainting: Random 3D patches of the input volume are masked, and the model learns to reconstruct the missing voxel intensities.
• Contrastive Learning: A variant uses a contrastive loss to pull representations of augmented views of the same sub-volume together while pushing apart representations from different volumes.
• Rotation Prediction: The model predicts the rotation applied to a sub-volume, learning orientation-aware features.

The core computational unit within each Swin Transformer block is the 3D window-based multi-head self-attention (W-MSA). The input 3D feature map is partitioned into non-overlapping cubic windows. Self-attention is computed strictly within each window, limiting the computational complexity to linear with respect to volume size.

• Window size: Typically 7x7x7 voxels.
• Shifted window (SW-MSA): In the subsequent block, the window partition is shifted by half the window size, creating new window boundaries that bridge the previous partition's gaps.
• Relative position bias: A learnable bias matrix B is added to the attention scores within each window to encode the relative 3D spatial relationships between tokens.

Swin UNETR has demonstrated state-of-the-art results on challenging public benchmarks, particularly for multi-organ segmentation in CT scans. On the BTCV (Beyond the Cranial Vault) Abdomen dataset, it achieved a mean Dice score exceeding 90% for 13 abdominal organs. For brain tumor segmentation on the BraTS challenge, it excels at segmenting heterogeneous tumor sub-regions (enhancing tumor, peritumoral edema, necrotic core).

• Robustness: The hierarchical design captures both the global anatomical context and fine local boundaries.
• Efficiency: Despite processing 3D volumes, the windowed attention mechanism keeps memory consumption manageable on modern GPUs.

An enhanced variant incorporates a contrastive pre-training framework to improve the encoder's feature learning. A 3D volume is passed through two different random augmentations (e.g., intensity scaling, rotation, cropping) to create a positive pair. The encoder processes both views, and a projection head maps the global representation to a latent space where a contrastive loss maximizes agreement between the positive pair while repelling negative examples from other volumes.

• Domain-specific augmentations: Tailored to medical imaging, including cutout, Gaussian noise, and gamma correction.
• Transfer learning: The pre-trained encoder provides a superior initialization, reducing the annotated data required for fine-tuning on downstream segmentation tasks.