Memory Efficiency

The core memory-saving mechanism. The massive, pre-trained foundation model (e.g., LLaMA 3, GPT-4) is loaded into VRAM in a read-only state. Its parameters are locked and do not receive gradient updates, eliminating the need to store optimizer states (like momentum) for billions of parameters. This is the single largest contributor to VRAM reduction, as optimizer states often consume 2-3x the memory of the weights themselves during full fine-tuning.

The small set of trainable parameters that represent the learned adaptation. In LoRA, these are the low-rank matrices A and B. Only these parameters require:

• Gradient computation during backpropagation.
• Optimizer state storage (e.g., for AdamW).
• Active updates during each training step. Because their count is often 0.1% to 1% of the base model's parameters, their memory footprint is negligible compared to full fine-tuning. They constitute the delta weights (ΔW) applied to the frozen base model.

The memory required to store intermediate layer activations during the forward pass for gradient calculation in the backward pass. This scales with:

• Batch size
• Sequence length
• Model hidden dimension Gradient checkpointing (or activation recomputation) is a critical technique where only a subset of activations are stored; the rest are recomputed during the backward pass. This trades compute for memory, often reducing activation memory by 60-70% at the cost of ~30% increased training time. It is essential for fitting long sequences.

The memory allocated for the optimizer's internal variables. For the Adam optimizer, this includes:

• First moment estimates (momentum)
• Second moment estimates (variance) These states are typically 32-bit floating point and are stored per trainable parameter. In full fine-tuning, this dominates VRAM usage. With LoRA, optimizer states are only needed for the tiny adapter matrices. For example, fine-tuning a 7B parameter model with LoRA (rank=8) may require optimizer states for only ~4M parameters, a reduction of over 99.9%.

A technique to load the frozen base model weights in a lower numerical precision to drastically reduce memory footprint. QLoRA uses 4-bit NormalFloat (NF4) quantization to load the model, reducing its memory requirement by ~4x (from 16-bit). Crucially, during forward and backward passes, weights are dequantized to 16-bit for computation, preserving performance. This allows fine-tuning of models like 65B parameter LLMs on a single 48GB GPU. The adapter matrices are typically kept in 16-bit precision.

A practical comparison of VRAM allocation for a 7B parameter model during fine-tuning on a 24GB GPU:

Full Fine-Tuning (Impossible on 24GB):

• Model Weights (16-bit): ~14 GB
• Optimizer States (Adam): ~28 GB
• Gradients: ~14 GB
• Activations: ~Variable, large TOTAL: >> 24 GB

LoRA Fine-Tuning (Feasible):

• Frozen Model Weights (16-bit): ~14 GB
• Adapter Params + Optimizer States: < 0.1 GB
• Gradients (for adapters): < 0.05 GB
• Activations (with checkpointing): ~4-8 GB TOTAL: ~18-22 GB

This approach strategically selects only a critical subset of layers or parameters to tune, based on sensitivity analysis or heuristics.

• Layer Selection: Often, only the attention query and value projections in transformer layers are tuned, as they are found to be most impactful for adaptation.
• DiffPruning: Learns a sparse binary mask over the original weights, applying updates only where the mask is 1.
• Memory Gain: By avoiding updates to FFN layers or key projections, the active parameter count can be reduced by 50-75% compared to applying LoRA to all linear layers.

These are system-level techniques that trade compute for memory, crucial for pushing the limits of model size on a given GPU.

• Gradient Checkpointing: Only stores activations for a subset of layers during the forward pass. The others are recomputed during the backward pass. Can reduce activation memory by ~60% at the cost of ~30% more compute time.
• Optimizer State Offloading: Moves the optimizer states (e.g., momentum and variance in Adam) from GPU VRAM to CPU RAM, transferring them back only for the update step. This is a core component of systems like DeepSpeed ZeRO-Offload.
• Combined Use: Essential for making techniques like QLoRA feasible on very large models, managing the memory of gradients and optimizer states for the small number of adapter parameters.

Parameter efficiency is the design principle of achieving high task performance while updating or adding only a minimal fraction of a model's total parameters. It is the core objective of PEFT methods like LoRA.

• Key Metric: The ratio of trainable parameters to total model parameters.
• Direct Impact: Directly reduces the size of optimizer states (e.g., Adam momentum and variance), which are a major component of GPU memory consumption during training.
• Example: Fine-tuning a 7B parameter model with LoRA (rank=8) may train only ~4M parameters, achieving >99.9% parameter efficiency.

Compute efficiency optimizes the usage of computational resources like FLOPs (floating-point operations) and GPU memory during training. It is closely tied to, but distinct from, memory efficiency.

• FLOPs Reduction: Methods like LoRA avoid computing gradients for the entire dense weight matrix, significantly reducing backward pass computation.
• Memory Bandwidth: Efficient methods minimize data movement between GPU memory and processors, a common bottleneck.
• Trade-off: The ultimate goal is to maximize task performance per unit of computational cost (e.g., performance per GPU-hour).

Gradient checkpointing (or activation checkpointing) is a memory optimization technique that trades compute for memory by selectively re-computing intermediate activations during the backward pass, rather than storing them all.

• Mechanism: Only a subset of layer outputs are stored during the forward pass; others are recomputed when needed for gradient calculation.
• Memory Savings: Can reduce activation memory by up to 70-80%, enabling the training of larger models or batch sizes.
• Synergy with PEFT: Often used in conjunction with LoRA to push the boundaries of feasible model size on limited VRAM.

Quantization reduces the numerical precision of model weights and activations (e.g., from 32-bit floating-point to 8-bit or 4-bit integers) to decrease memory footprint and accelerate computation.

• Weight Quantization: Storing the frozen base model in lower precision (e.g., INT4) drastically reduces its memory footprint.
• QLoRA: A seminal method that combines 4-bit quantization of the base model with LoRA adapters, enabling fine-tuning of 65B parameter models on a single 48GB GPU.
• Inference Speed: Quantized models also require less memory bandwidth, leading to faster inference times on compatible hardware.

Optimizer state memory refers to the auxiliary variables maintained by an optimization algorithm (like Adam) for each trainable parameter, which often consume the majority of GPU memory during training.

• Components: For Adam, this includes two moments (momentum and variance) per parameter, typically stored in 32-bit precision.
• Memory Cost: Can be 2-3x the size of the gradients themselves. For a 7B model, full fine-tuning requires ~28GB just for optimizer states (in FP32).
• PEFT Advantage: By making only a tiny fraction of parameters trainable, LoRA reduces optimizer state memory proportionally (e.g., from 28GB to ~16MB).

Mixed precision training uses lower-precision data types (like FP16/BF16) for most operations to save memory and increase throughput, while keeping a master copy of weights in higher precision (FP32) for numerical stability.

• Memory Reduction: FP16 tensors use half the memory of FP32 tensors.
• Speed Increase: Lower precision operations can be executed faster on modern GPU tensor cores.
• Standard Practice: Combined with PEFT, it is a foundational technique for memory-efficient fine-tuning. Libraries like PyTorch's AMP (Automatic Mixed Precision) automate this process.