DeepSpeed

The ZeRO (Zero Redundancy Optimizer) family eliminates memory redundancy across data-parallel processes. Instead of replicating model states on every GPU, ZeRO partitions optimizer states, gradients, and parameters across devices.

• ZeRO-1: Partitions optimizer states only (4x memory reduction)
• ZeRO-2: Adds gradient partitioning (8x reduction)
• ZeRO-3: Partitions all model states including parameters, enabling models with trillions of parameters

Each stage communicates only the necessary data on demand, scaling super-linearly with GPU count.

ZeRO-Offload transparently moves optimizer states and computations from GPU memory to CPU memory and compute. This enables training models 10x larger than GPU memory alone would allow.

• Offloads fp32 optimizer states and momentum buffers to CPU RAM
• Keeps fp16 parameters and forward/backward passes on GPU
• CPU computes the optimizer step while GPU proceeds to the next iteration
• Achieves near-GPU-bound throughput through computation-communication overlap

A single GPU can now train models previously requiring multiple high-memory accelerators.

ZeRO-Infinity extends the offload paradigm to NVMe SSDs, treating disk as an extension of the memory hierarchy. This breaks the GPU memory wall entirely for training.

• Infinity Offload Engine: Moves parameters, gradients, and optimizer states to NVMe
• Bandwidth-Centric Partitioning: Optimizes data placement based on bandwidth hierarchy (GPU HBM > CPU DRAM > NVMe)
• Overlap Centric Design: Hides data movement latency behind computation

Enables training models with tens of trillions of parameters on a single GPU by leveraging cheap, abundant NVMe storage.

DeepSpeed integrates three complementary parallelism strategies into a unified framework for scaling to thousands of GPUs.

• Data Parallelism: Replicates the model across GPUs, each processing different input batches
• Tensor Parallelism: Splits individual weight matrices across GPUs using model parallelism within a single transformer layer
• Pipeline Parallelism: Divides model layers sequentially across GPU groups, passing activations between stages

The 3D Parallelism engine automatically determines the optimal combination of these strategies based on model architecture, batch size, and cluster topology.

A specialized inference system designed for low-latency, high-throughput serving of large transformer models in production.

• Multi-GPU Inference: Leverages tensor parallelism to distribute model layers across GPUs
• Inference-Adapted ZeRO: Applies ZeRO partitioning specifically optimized for the inference memory footprint
• Kernel Fusion: Fuses multiple CUDA operations into single kernels to reduce kernel launch overhead
• Quantization Support: Integrates with INT8 and other quantization techniques for further latency reduction

Delivers up to 7.3x higher throughput and 2.4x lower latency compared to standard PyTorch inference.

DeepSpeed includes a composable compression toolkit that reduces model size and communication overhead during training.

• Weight Quantization: Reduces parameter precision to INT8 or INT4
• Activation Compression: Compresses activations before communication in pipeline-parallel setups
• Gradient Compression: Applies sparsification or quantization to gradients before all-reduce operations
• Layer Reduction: Prunes entire layers or attention heads based on importance scoring

These techniques reduce communication bandwidth by up to 5x with negligible accuracy loss, critical for bandwidth-constrained clusters.