Vision Transformer (ViT)

The input image x ∈ R^(H×W×C) is reshaped into a sequence of flattened 2D patches x_p ∈ R^(N×(P²·C)), where (P, P) is the patch resolution and N = HW/P² is the effective sequence length. Each patch is linearly projected to a D-dimensional latent vector via a trainable embedding matrix E ∈ R^((P²·C)×D). A learnable [class] token embedding is prepended to the sequence, serving as the final image representation for classification. Standard 1D learnable position embeddings are added to retain spatial information.

The core computational unit enabling global context. Input tokens are projected into Query (Q), Key (K), and Value (V) matrices. Attention weights are computed as the scaled dot-product softmax of Q and K^T: Attention(Q,K,V) = softmax(QK^T/√d_k)V. Multi-head attention runs this operation in parallel across h subspaces, allowing the model to jointly attend to information from different representation subspaces. This mechanism has quadratic complexity O(N²·D) with respect to sequence length.

ViT stacks L identical Transformer encoder layers. Each block consists of two sub-layers:

• Multi-Head Self-Attention with a residual connection.
• MLP Block: A two-layer feed-forward network with GELU activation: FFN(x) = W₂ · GELU(W₁ · x + b₁) + b₂. The hidden dimension is typically 4× D. Layer Normalization (LN) is applied before each sub-layer (Pre-LN), and residual connections wrap each sub-layer. This Pre-LN design improves training stability in deep architectures.

Since self-attention is permutation-invariant, explicit positional information must be injected. ViT uses learned 1D position embeddings added directly to the patch embeddings. Each spatial position receives a unique D-dimensional vector learned during training. This contrasts with fixed sinusoidal encodings used in NLP Transformers. The model learns to encode spatial relationships, though it lacks a strong inductive bias for translation equivariance, requiring large-scale pre-training (e.g., JFT-300M) to match CNNs.

The final image representation is extracted from the output of the [class] token at the last encoder layer. This vector z_L^0 is passed through a single linear classification head (a fully-connected layer with no hidden layers) during pre-training. During fine-tuning, this head is replaced with a task-specific zero-initialized feed-forward layer. The architecture avoids any 2D-specific pooling operations, treating the image purely as a sequence of tokens from input to output.

Instead of raw image patches, the input sequence can be formed from feature maps extracted by a CNN backbone. A standard ResNet processes the image, and the output feature map (e.g., 14×14 from ResNet50) is flattened into tokens. This hybrid CNN-Transformer approach injects convolutional inductive biases (locality, translation equivariance) into the early stages, enabling competitive performance on smaller datasets like ImageNet-1k without requiring massive pre-training datasets.

ViTs process gigapixel pathology slides by treating tissue patches as token sequences. Multiple Instance Learning (MIL) aggregates patch-level representations into a slide-level diagnosis.

• Detects metastatic breast cancer in lymph node biopsies
• Grades prostate cancer Gleason scores automatically
• Handles variable input sizes without architectural changes

Key advantage: Self-attention captures morphological context across distant tissue regions that CNNs with limited receptive fields miss.

Hierarchical ViTs like Swin UNETR perform dense voxel-level segmentation of CT and MRI volumes. The U-shaped encoder-decoder design with skip connections preserves fine spatial detail.

• Brain tumor sub-region segmentation (BraTS benchmark)
• Liver and hepatic vessel delineation for surgical planning
• Multi-organ segmentation for radiation therapy contouring

Self-attention across axial, coronal, and sagittal planes captures 3D anatomical context that 2D slice-based methods inherently lack.

ViTs classify and localize pathologies in frontal chest radiographs by attending globally to both lung fields simultaneously. DINO self-supervised pre-training produces attention maps that segment organs without pixel-level labels.

• Pneumonia, pneumothorax, and pleural effusion triage
• Cardiomegaly detection via heart silhouette analysis
• Multi-label classification across 14+ finding types (CheXpert)

Clinical impact: Global context prevents false negatives from pathologies outside typical ROI windows.

ViTs analyze full-field digital mammograms by dividing high-resolution images into patches and learning subtle textural and architectural distortions indicative of malignancy.

• Detects masses, microcalcifications, and architectural distortions
• Reduces false-positive recall rates by 5-9%
• Matches or exceeds radiologist standalone sensitivity

Self-attention compares symmetric breast tissue patterns across views (CC and MLO), mimicking the radiologist's bilateral comparison workflow.

ViTs process color fundus photographs and optical coherence tomography (OCT) scans to detect diabetic retinopathy, glaucoma, and age-related macular degeneration.

• Grades diabetic retinopathy severity on the International Clinical Scale
• Measures cup-to-disc ratio for glaucoma assessment
• Identifies drusen and geographic atrophy in AMD

Global attention captures vascular patterns and lesion distributions across the entire retinal field, critical for staging systemic disease progression.

ViTs serve as the imaging backbone in architectures that fuse radiology with clinical text and genomics. Contrastive Language-Image Pre-training (CLIP) enables zero-shot retrieval of relevant priors.

• Combines CT volumes with electronic health record text
• Aligns pathology images with genomic mutation profiles
• Enables text-based querying of image databases

Architecture: Separate ViT and text encoders project into a shared embedding space where matched pairs have high cosine similarity, enabling cross-modal reasoning.