Vision Transformer (ViT)

The foundational step where a gigapixel whole-slide image (WSI) is divided into a grid of fixed-size 2D patches (e.g., 16x16 pixels). Each patch is linearly projected into a flat vector, creating a sequence of visual tokens analogous to words in a sentence. A learnable [class] token is prepended to the sequence, and positional embeddings are added to retain spatial context, converting unstructured pixel data into a structured input for the Transformer encoder.

The core computational engine that allows every patch to interact with every other patch globally. Unlike convolutional neural networks (CNNs) with limited receptive fields, MHSA computes attention weights between all token pairs in a single layer. This mechanism directly captures long-range spatial dependencies—such as the relationship between a distant tumor nest and a tertiary lymphoid structure—which is critical for understanding complex tissue architecture in tasks like Gleason grading.

Following the attention mechanism, each token representation is passed through a position-wise feed-forward network consisting of two linear transformations with a GELU non-linearity. This block operates independently on each token, introducing non-linear feature transformations and increasing the model's representational capacity. The MLP typically has a hidden dimension expansion ratio of 4x, enabling complex feature interactions within individual patch representations.

Stabilizing components applied before each sub-layer (pre-norm architecture). Layer Normalization standardizes inputs across the feature dimension, reducing internal covariate shift and accelerating training convergence. Residual skip connections wrap both the MHSA and MLP blocks, allowing gradients to flow directly through the network during backpropagation. This design enables the stable training of very deep ViT architectures (e.g., ViT-Large with 24 layers) on large-scale histology datasets.

The final [class] token output from the Transformer encoder serves as a holistic image representation, aggregating information from all patches through self-attention. This vector is passed through a simple linear classifier for slide-level prediction tasks such as tumor vs. normal classification. Alternatively, the patch-level output tokens can be reshaped into a feature map for dense prediction tasks like semantic segmentation of tissue regions, making ViT a versatile backbone for both slide-level and pixel-level analysis.

ViTs excel when pre-trained on massive unlabeled histology datasets using self-supervised learning (SSL) objectives before fine-tuning on smaller annotated cohorts. Key paradigms include:

• Masked Image Modeling (MIM): Randomly masking a high proportion of patches (e.g., 75%) and training the model to reconstruct the missing content, forcing it to learn meaningful tissue representations.
• DINO (Self-Distillation with No Labels): A teacher-student framework where the model learns by matching outputs from different augmented views of the same image, producing attention maps that naturally segment histological structures without pixel-level supervision.

The core mathematical operation that powers Vision Transformers. Self-attention computes a weighted sum of all input patch embeddings, allowing the model to dynamically assess the relevance of every image region to every other region. Unlike the fixed receptive fields of convolutional neural networks, self-attention captures long-range spatial dependencies across an entire whole-slide image. In pathology, this enables a ViT to relate a tumor nest in one corner of a gigapixel image to stromal reaction on the opposite side, learning global tissue architecture patterns critical for accurate grading.

The process of dividing an input image into a grid of fixed-size, non-overlapping patches and linearly projecting each into a flat vector. This tokenization step converts raw pixels into a sequence of patch embeddings that the Transformer can process. For a 256x256 pixel histology region at 16x16 patch size, the model creates 256 tokens. Each embedding captures local texture and nuclear morphology, while the subsequent self-attention layers integrate these local features into a global representation of tissue architecture.

A mechanism that injects spatial location information into patch embeddings, since self-attention is inherently permutation-invariant and has no notion of spatial order. Vision Transformers typically use learned positional embeddings that are added to patch tokens before entering the Transformer encoder. These encodings allow the model to distinguish the top-left stroma from the bottom-right tumor, preserving the 2D spatial structure of tissue. Without positional encoding, a ViT would treat a histology image as a bag of unordered patches, losing critical architectural context.

An extension of self-attention that runs multiple attention operations in parallel, each with its own learned projection matrices. Each attention head specializes in capturing different relational patterns: one head might focus on nuclear density, another on glandular formation, and a third on immune cell clustering. The outputs are concatenated and projected back to the original dimension. In digital pathology, multi-head attention enables simultaneous analysis of cytological detail, architectural patterns, and spatial context within a single Transformer layer.

A variant architecture, such as the Swin Transformer, that computes self-attention within local shifted windows rather than globally across all patches. This design produces multi-scale feature maps analogous to convolutional pyramids, making it efficient for dense prediction tasks like semantic segmentation of tumor regions. By limiting attention to local windows and merging patches at deeper layers, hierarchical ViTs achieve linear computational complexity relative to image size, enabling processing of larger histology patches without the quadratic cost of standard ViTs.

A training paradigm where Vision Transformers learn general-purpose visual features from massive unlabeled histology datasets before fine-tuning on specific diagnostic tasks. Methods like Masked Autoencoding (MAE) randomly mask a high proportion of image patches and train the ViT to reconstruct the missing content, forcing it to learn tissue morphology, staining patterns, and cellular relationships. Pre-trained foundation models like UNI and Virchow leverage this approach on millions of whole-slide images, producing robust patch encoders that transfer effectively to downstream biomarker quantification tasks.