Vision Transformer (ViT)

The input gigapixel whole slide image is divided into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels). Each patch is linearly flattened into a 1D vector and projected into a lower-dimensional embedding space via a trainable linear layer. A learnable [class] token is prepended to this sequence of patch embeddings, and 1D positional encodings are added to retain spatial information, converting the 2D image into a 1D sequence suitable for the Transformer encoder.

Unlike CNNs with their local receptive fields, ViT's core is the multi-head self-attention (MSA) mechanism. MSA computes weighted relationships between every patch in the sequence, allowing the model to directly model long-range dependencies from the earliest layers.

• Query, Key, Value (QKV): Each patch embedding is projected into Q, K, and V vectors.
• Attention Weights: The dot product of Q and K determines how much each patch "attends to" every other patch.
• Pathology Relevance: This is critical for understanding tissue architecture, where a diagnostic region's significance depends on its relationship to distant stromal or necrotic areas.

The sequence of patch embeddings is processed by a stack of L identical Transformer encoder blocks. Each block consists of two core sub-layers with residual connections:

• Multi-Head Self-Attention (MSA): Aggregates global contextual information.
• Multi-Layer Perceptron (MLP): A two-layer feed-forward network with a GELU non-linearity, applied identically to each position. Layer Normalization (LayerNorm) is applied before each sub-layer (pre-norm configuration), and dropout is used for regularization. This deep stack iteratively refines patch representations based on the full image context.

The final representation of the prepended [class] token from the last encoder block serves as the aggregate image representation. This vector is passed to a simple MLP classification head for slide-level diagnosis.

• Pre-Training Imperative: ViTs are data-hungry and lack the inductive biases (translation equivariance, locality) of CNNs. They typically require pre-training on massive datasets (e.g., JFT-300M, ImageNet-21k) or via self-supervised learning (SSL) on unlabeled histology data before fine-tuning on smaller, annotated pathology datasets.
• Fine-Tuning: The pre-trained model is adapted to a downstream task (e.g., Gleason grading) by replacing the classification head and training at a higher resolution.

Standard ViT produces a single-scale feature map, which is suboptimal for dense prediction tasks like tissue segmentation. Hierarchical Vision Transformers, such as the Swin Transformer, address this by computing self-attention within shifted local windows and merging patches in deeper layers to build a pyramidal feature hierarchy.

• Swin Transformer: Produces multi-scale feature maps identical in resolution to CNN backbones (e.g., ResNet), making it a drop-in replacement for architectures like Mask R-CNN.
• Benefit: This design captures both fine-grained cellular details and global tissue architecture, achieving state-of-the-art performance in both classification and segmentation tasks.

A key advantage of the attention mechanism is its inherent explainability. The attention weights computed in the final encoder blocks can be visualized as a heatmap over the original image, highlighting the patches the model deemed most salient for its prediction.

• Clinical Audit: Pathologists can inspect these attention maps to verify that the model's diagnosis is based on morphologically relevant tissue regions (e.g., tumor epithelium) and not on artifacts or background.
• Contrast with CNNs: This provides a more direct and intuitive form of visual explanation compared to post-hoc gradient-based methods like Grad-CAM, fostering greater clinical trust.

The foundational pre-processing step where a gigapixel Whole Slide Image is tessellated into fixed-size patches (e.g., 16×16 or 256×256 pixels). Each patch is flattened and linearly projected into a feature embedding vector, forming the input token sequence for the transformer. This converts raw pixel data into a 1D sequence compatible with the self-attention mechanism, preserving spatial context through learned positional encodings.

The core mechanism enabling ViTs to model long-range dependencies across an entire slide. Unlike CNNs with limited receptive fields, multi-head self-attention computes pairwise relationships between every patch, allowing the model to weigh the importance of distant tissue regions. This captures architectural patterns like tumor-stroma interactions and immune cell distribution that are critical for Gleason grading and TMB prediction.

A mechanism that injects spatial information into the permutation-invariant transformer architecture. Learned positional embeddings encode the 2D coordinates of each patch, preserving the topological layout of tissue. This is essential for distinguishing architecturally distinct regions—such as invasive front versus tumor core—and enables the model to understand spatial hierarchies without convolutional inductive biases.

A comparative framework highlighting architectural trade-offs:

• CNNs: Strong local feature extraction, translation equivariance, efficient on limited data
• ViTs: Global context modeling, superior scalability with data volume, fewer inductive biases
• Hybrid approaches: CNN patch feature extraction followed by transformer aggregation, combining local texture analysis with global architectural reasoning ViTs excel when pre-trained on massive histology datasets via self-supervised learning, often outperforming CNNs on slide-level classification tasks.

Advanced ViT variants like Swin Transformer and Hiera that compute self-attention within local windows and merge patches across layers. This creates multi-scale feature pyramids analogous to CNNs, efficiently handling the gigapixel scale of WSIs. Hierarchical designs reduce the quadratic complexity of global attention while maintaining the ability to model both cellular-level details and tissue-level architecture, critical for tumor microenvironment analysis.

The dominant paradigm combining Multiple Instance Learning with transformer backbones. A ViT processes patches to produce instance-level embeddings, which are then aggregated via attention-based pooling for slide-level classification. Frameworks like CLAM and TransMIL use transformer layers to model inter-patch relationships before final aggregation, enabling weakly supervised learning from only slide-level labels while capturing spatial tissue architecture.