ZeRO Optimization

ZeRO operates in three sequential optimization stages, each partitioning a different component of the model state to eliminate data-parallel redundancy:

• Stage 1 (ZeRO-1): Partitions optimizer states across GPUs, reducing memory by up to 4x. Each GPU holds only a fraction of the Adam optimizer's momentum and variance buffers.
• Stage 2 (ZeRO-2): Additionally partitions gradients, achieving up to 8x memory reduction. Gradients are reduced-scattered to their owning GPU during backpropagation.
• Stage 3 (ZeRO-3): Partitions model parameters themselves, distributing the entire model state across all devices. Parameters are fetched on-demand during forward and backward passes, enabling models with trillions of parameters to fit on commodity hardware.

For legal pre-training, ZeRO-3 is essential when working with billion-parameter models and long sequence lengths required for multi-document reasoning.

When GPU memory remains insufficient even with partitioning, ZeRO-Offload extends the optimization hierarchy by transparently moving data to external memory tiers:

• CPU Offload: Optimizer states and gradients are offloaded to CPU DRAM during computation. The Adam optimizer, which consumes the majority of memory during pre-training, runs efficiently on the CPU while forward and backward passes execute on the GPU.
• NVMe Offload: For extreme-scale models, ZeRO-Infinity extends offloading to NVMe SSDs, treating them as an extension of GPU memory. A prefetching scheduler overlaps data movement with computation to hide I/O latency.
• One-Step Delayed Parameter Update (DPU): A novel scheduling technique that overlaps CPU optimizer computation with the next training step's forward pass, effectively hiding the CPU-GPU communication overhead.

This hierarchy enables legal teams with limited GPU clusters to pre-train domain-adapted models that would otherwise require supercomputing-scale infrastructure.

ZeRO achieves near-linear scaling across GPUs by overlapping communication with computation, a critical feature for distributed legal pre-training:

• Reduce-Scatter: Instead of an all-reduce operation that duplicates gradients on every GPU, ZeRO uses a reduce-scatter collective that partitions gradients directly to their owning device, halving communication volume.
• Communication Overlap: Gradient reduction is overlapped with the backward pass computation. As soon as gradients for a layer are computed, their reduction begins immediately, hiding network latency behind computation.
• Bandwidth Optimization: ZeRO dynamically fuses small parameter groups to maximize network utilization, critical for the irregular tensor sizes common in transformer architectures.

For legal domain-adaptive pre-training (DAPT), where training corpora may span terabytes of case law and statutes, this communication efficiency directly translates to reduced wall-clock training time and lower cloud compute costs.

ZeRO is designed to work seamlessly with mixed-precision training using BFloat16 and FP16 formats, compounding memory savings for legal model pre-training:

• FP16 Master Weights: While forward and backward passes use half-precision for speed, ZeRO maintains a full-precision FP32 master copy of weights partitioned across GPUs, preserving numerical stability during updates.
• Dynamic Loss Scaling: Integrated loss scaling prevents gradient underflow in half-precision, automatically adjusting the scale factor to maintain training stability on long legal documents with sparse gradients.
• BFloat16 Compatibility: ZeRO supports BFloat16, which preserves the same dynamic range as FP32, eliminating the need for loss scaling entirely and simplifying the training of large legal models.

Combined with FlashAttention for long-sequence processing, ZeRO and mixed-precision enable the pre-training of legal models on documents exceeding 8,192 tokens—essential for full contract analysis and multi-document case synthesis.

ZeRO is commonly paired with activation checkpointing (also called gradient checkpointing) to further reduce memory consumption during legal model pre-training:

• Activation Memory Trade-off: Instead of storing all intermediate activations for backpropagation, only layer boundaries are saved. During the backward pass, activations are recomputed on-the-fly, trading a ~33% increase in computation for a 5-10x reduction in activation memory.
• Partitioned Activations: ZeRO-3 extends checkpointing by partitioning saved activation checkpoints across GPUs, ensuring that even the recomputation buffers do not become a memory bottleneck.
• Selective Checkpointing: For transformer-based legal models, only attention and feed-forward layer outputs are checkpointed, preserving the maximum memory savings while minimizing recomputation overhead.

This combination is essential for training on the long sequence lengths required by legal documents, where activation memory scales quadratically with sequence length in the self-attention mechanism.

ZeRO-3 introduces a sophisticated on-demand parameter fetching mechanism that is transparent to the model architecture:

• All-Gather on Demand: Before each transformer layer's forward pass, an all-gather collective retrieves the required parameter partition from all GPUs. After the layer executes, parameters are released, freeing memory for the next layer.
• Prefetching Scheduler: A look-ahead scheduler anticipates which parameters will be needed next and initiates their fetch while the current layer is still computing, hiding communication latency behind computation.
• Memory Defragmentation: ZeRO-3 includes a memory defragmentation system that coalesces freed parameter buffers, preventing the memory fragmentation that can cause out-of-memory errors during long training runs.

For legal Mixture of Experts (MoE) architectures, where different experts activate for different legal sub-domains, ZeRO-3's dynamic fetching ensures that only active expert parameters consume GPU memory at any given moment.