Low-Rank Adaptation (LoRA)

A parameter-efficient adaptation technique that prepends a small set of learnable continuous embeddings to the input sequence of a frozen Vision Transformer. Unlike LoRA, which modifies internal weight matrices, VPT steers the model by injecting task-specific tokens into the input space only.

• VPT-Shallow: Prompts inserted into the first Transformer layer only
• VPT-Deep: Prompts inserted at every layer, matching or exceeding full fine-tuning accuracy on medical imaging benchmarks
• Typically requires tuning fewer than 1% of total model parameters

A model compression technique where a smaller student model is trained to mimic the softened output logits or intermediate feature representations of a larger, pre-trained teacher model. In medical imaging, this is often used to transfer diagnostic knowledge from a massive ViT to a compact deployable model.

• Soft targets: Use a temperature-scaled softmax to reveal inter-class relationships learned by the teacher
• Feature-based distillation: Align intermediate attention maps between teacher and student
• Enables deployment of high-accuracy diagnostic models on edge hardware in clinical settings

A memory optimization technique that trades compute for memory by discarding intermediate activations during the forward pass and recomputing them on demand during the backward pass. This is critical when fine-tuning large Vision Transformers on high-resolution 3D medical volumes.

• Reduces peak GPU memory consumption by up to 60%
• Selectively applied to attention and feed-forward blocks
• Enables LoRA fine-tuning of ViT-Large on full-resolution CT scans without model parallelism
• Increases training time by approximately 20-30% due to recomputation overhead

A training paradigm that uses lower-precision 16-bit floating-point formats (FP16 or BF16) for most forward and backward pass operations while maintaining a master copy of weights in 32-bit precision. When combined with LoRA, this dramatically reduces the memory footprint of medical model adaptation.

• FP16: IEEE half-precision, requires loss scaling to prevent gradient underflow
• BF16: Brain floating-point format with the same exponent range as FP32, eliminating the need for loss scaling
• Reduces memory consumption by nearly 50% and accelerates matrix multiplications on Tensor Cores
• Essential for fine-tuning on large 3D medical imaging datasets within single-GPU memory budgets

An optimization algorithm that simultaneously minimizes loss value and loss sharpness by seeking parameters in neighborhoods of uniformly low loss. Applied during LoRA fine-tuning, SAM improves generalization on out-of-distribution medical imaging data from different scanner vendors.

• Performs a two-step update: ascent to find worst-case perturbation, then descent from that point
• Flattens the loss landscape, reducing sensitivity to weight perturbations
• Demonstrated to improve diagnostic accuracy on external validation cohorts by 2-5% over standard AdamW
• Increases per-step compute cost by approximately 2x due to the additional forward-backward pass

A regularization technique that randomly drops entire residual blocks or Transformer layers during training, forcing the network to learn robust representations that do not depend on any single computational path. When combined with LoRA adapters, it prevents overfitting to small medical datasets.

• Each layer is dropped with a probability that increases linearly with depth
• At inference time, all layers are active with scaled weights
• Reduces training time by approximately 25% while improving generalization
• Particularly effective for pathology foundation models fine-tuned on limited whole slide image annotations