Token Merging (ToMe)

A memory optimization technique that trades compute for memory. During the forward pass, intermediate activations are discarded rather than stored. During the backward pass, these activations are recomputed on demand from the nearest saved tensors. This allows training larger Vision Transformer models on high-resolution medical images with limited GPU memory. While ToMe reduces the computational graph size directly, gradient checkpointing addresses memory constraints without altering the model's forward computation.

A sparse attention mechanism where each query attends only to a small, learned set of sampling locations around a reference point, rather than all tokens. This enables efficient multi-scale feature aggregation with linear complexity relative to image resolution. Unlike ToMe, which merges redundant tokens globally, deformable attention avoids computing attention over irrelevant regions entirely, making it particularly effective for object detection in high-resolution radiology scans.

A training paradigm that uses lower-precision 16-bit floating-point (FP16 or BF16) for most arithmetic operations while maintaining a master copy of weights in 32-bit precision. This reduces memory consumption by nearly half and accelerates computation on modern tensor-core GPUs. When combined with ToMe during inference, mixed precision further compresses the model's runtime footprint, enabling deployment of diagnostic Vision Transformers on edge hardware with limited resources.

A regularization technique that randomly drops entire residual blocks during training, forcing the network to learn robust representations that do not depend on any single computational path. This reduces training time and acts as an implicit model ensemble. While ToMe permanently merges tokens to reduce inference cost, stochastic depth provides a complementary benefit during the training phase by preventing co-adaptation of layers, which can improve the model's tolerance to token reduction at inference.

A vision backbone that applies a bidirectional state space model (SSM) to sequences of image patches, offering linear-time complexity as an alternative to the quadratic self-attention of Vision Transformers. Unlike ToMe, which retrofits efficiency onto a standard ViT, Vim is architected from the ground up for linear scaling. For medical imaging tasks requiring processing of gigapixel whole slide images, Vim represents a competing paradigm to token reduction strategies.