Gradient Accumulation

Gradient accumulation enables training with an effective batch size far larger than what fits in GPU memory. Instead of computing gradients on a full batch of 256 sequences at once, the process runs 8 micro-batches of 32, accumulating gradients across forward and backward passes before a single optimizer step. This is critical for genomic models where large batch statistics stabilize training of DNA language models on variable-length sequences.

Accumulating gradients over k micro-batches is mathematically identical to processing a single batch k times larger, provided the loss function is averaged over samples. The gradient of the summed loss equals the sum of individual gradients. However, layers with Batch Normalization break this equivalence because running statistics are computed per micro-batch. Genomic models typically use Layer Normalization or Instance Normalization to avoid this discrepancy.

Standard training loops call optimizer.zero_grad(), loss.backward(), and optimizer.step() in sequence. With gradient accumulation:

• Call loss.backward() for each micro-batch without zeroing gradients
• Gradients accumulate in .grad attributes
• Call optimizer.step() only after accumulation_steps iterations
• Then call optimizer.zero_grad() to reset

Frameworks like Hugging Face Trainer and PyTorch Lightning support this natively via an accumulation_steps parameter.

Gradient accumulation trades training time for memory efficiency. Processing 8 micro-batches sequentially takes roughly 8x longer per optimizer step than a single large batch, but peak GPU memory usage remains at the micro-batch level. This is essential when training genomic transformer models with long context windows (e.g., 8k+ nucleotide tokens) on GPUs with limited VRAM, such as consumer-grade hardware or shared clusters.

When combined with Distributed Data Parallelism (DDP), gradient accumulation requires careful synchronization. Gradients are accumulated locally on each GPU across micro-batches, and the all-reduce operation occurs only during the final optimizer.step() call. This reduces communication overhead proportionally to the accumulation steps. For ZeRO-optimized genomic model training, accumulation works alongside parameter partitioning to further minimize per-GPU memory.

When increasing the effective batch size via gradient accumulation, the learning rate must often be scaled to maintain training stability. The linear scaling rule suggests multiplying the learning rate by the accumulation factor. However, for very large effective batches (e.g., 4096+), LARS or LAMB optimizers with layer-wise adaptive rates are preferred. Genomic models using AdamW typically require warmup and cosine decay schedules when scaling batch sizes.

A complementary memory-saving technique that trades compute for memory. Instead of storing all intermediate activations for the backward pass, it discards them during the forward pass and recomputes them on-the-fly during backpropagation. When combined with gradient accumulation, this allows for extremely large effective batch sizes on memory-constrained hardware, which is critical for long-sequence DNA models.

A method that uses both 16-bit (FP16/BFLOAT16) and 32-bit (FP32) floating-point formats during training. A master copy of weights is kept in FP32 for numerical stability, while forward and backward passes execute in half-precision for speed. This directly reduces the memory footprint of the micro-batches used in gradient accumulation, allowing more accumulations before hitting GPU limits.

A memory optimization strategy from DeepSpeed that partitions model states (optimizer states, gradients, and parameters) across data-parallel processes. Unlike gradient accumulation which simulates a larger batch on a single device, ZeRO eliminates memory redundancy across devices. The two techniques are often combined: ZeRO partitions the model to fit it, and gradient accumulation simulates the global batch size.

The standard paradigm where a model is replicated across multiple GPUs, each processing a distinct subset of data. Gradients are synchronized via an all-reduce operation before each weight update. Gradient accumulation is orthogonal to this: it reduces the frequency of these expensive communication steps by accumulating local gradients over multiple micro-batches before triggering the global all-reduce.

An IO-aware exact attention algorithm that minimizes reads/writes between high-bandwidth memory (HBM) and on-chip SRAM. By tiling the attention computation, it reduces the memory footprint of the attention mechanism from O(N²) to O(N). This frees up significant GPU memory for larger micro-batch sizes, directly reducing the number of accumulation steps needed for a target effective batch size in genomic transformer models.

A critical implementation detail when using gradient accumulation with Batch Normalization (BatchNorm) layers. BatchNorm computes mean and variance statistics over the current micro-batch, not the accumulated effective batch. This discrepancy can destabilize training. The standard fix is to use SyncBatchNorm (synchronized across devices) or replace BatchNorm with LayerNorm or GroupNorm, which compute statistics independently of batch size.