ZeRO Optimization

ZeRO progressively partitions training state across three optimization stages, each reducing memory footprint further:

• Stage 1 (Optimizer State Partitioning): Shards optimizer states (e.g., Adam momentum and variance) across GPUs, reducing memory by 4x with no communication overhead increase
• Stage 2 (Gradient Partitioning): Adds gradient sharding on top of Stage 1, reducing memory by 8x with minimal communication overhead
• Stage 3 (Parameter Partitioning): Shards model parameters themselves, achieving memory reduction linear with the number of GPUs (Nx reduction for N GPUs) at the cost of modest inter-GPU communication

For a 10-billion-parameter genomic foundation model, Stage 3 can reduce per-GPU memory from over 120 GB to under 15 GB.

ZeRO-Offload extends memory capacity beyond GPU limits by transparently moving optimizer states and gradients to CPU memory or NVMe storage during training:

• CPU Offload: Moves optimizer states and fp32 master weights to CPU RAM, keeping only fp16 parameters and gradients on GPU. Achieves up to 10x larger model training on the same hardware
• NVMe Offload: Extends offload to high-speed NVMe SSDs for extreme model scales, enabling training of models with over 1 trillion parameters on a single GPU
• One-Step Delayed Parameter Update: A novel scheduling technique that overlaps CPU computation with GPU computation, hiding data transfer latency

This is critical for genomic institutions with limited GPU clusters who need to train large DNA language models.

ZeRO-Infinity extends the ZeRO family to support training of models with trillions of parameters—a scale relevant for genomic foundation models that must capture the full diversity of human genetic variation:

• Bandwidth-Centric Partitioning: Intelligently partitions model states across GPU, CPU, and NVMe memory tiers based on bandwidth requirements, not just capacity
• Overlap-Centric Communication: Aggressively overlaps communication with computation across all memory tiers to hide data movement latency
• Memory-Centric Tiling: Breaks large operators into smaller tiles that fit within available GPU memory, enabling forward and backward passes on models that exceed GPU memory by orders of magnitude

This architecture enables training of genomic models with up to 32 trillion parameters on 512 NVIDIA V100 GPUs.

ZeRO minimizes the communication overhead introduced by parameter sharding through aggressive overlapping of communication and computation:

• Bucketed All-Gather: Groups parameter shards into buckets and initiates all-gather communication for the next bucket while computing on the current bucket
• Reduce-Scatter Overlap: Overlaps gradient reduce-scatter operations with the backward pass computation of subsequent layers
• Prefetch Scheduling: Prefetches parameters required for upcoming computation steps, hiding network latency behind ongoing GPU operations

For genomic sequence models with long context windows (e.g., 100k+ base pairs), these overlap strategies maintain near-linear scaling efficiency even at 1,000+ GPUs.

Understanding ZeRO's memory savings requires analyzing the memory breakdown of training a typical genomic transformer with Adam optimizer:

• Model Parameters (fp16): 2 bytes × P parameters
• Gradients (fp16): 2 bytes × P parameters
• Optimizer States (fp32): 12 bytes × P parameters (fp32 master weights + momentum + variance)
• Activations: Variable, dependent on batch size, sequence length, and hidden dimension

Example: A 20B-parameter DNA language model requires ~320 GB just for parameters, gradients, and optimizer states. With ZeRO Stage 3 across 8 GPUs, this drops to ~40 GB per GPU, leaving room for activations and batch data. For long genomic sequences (100k tokens), activation memory often dominates, making ZeRO's parameter reduction essential.

ZeRO optimization integrates into production genomic training pipelines through DeepSpeed, Microsoft's open-source deep learning optimization library:

• Configuration-Driven: Memory partitioning strategy is specified via a JSON config file, enabling experimentation without code changes
• Checkpoint Consolidation: Automatically consolidates sharded checkpoints into a single unified model for inference deployment
• Elastic Training: Supports dynamic addition and removal of GPUs during training, critical for preemptible cloud instances used in genomic workloads
• Model Registry Compatibility: Sharded checkpoints include metadata for version tracking, enabling reproducible genomic model training across heterogeneous clusters

This configuration-driven approach allows bioinformatics teams to switch between ZeRO stages based on available hardware without modifying model architecture.