Model Parallelism

Partitions model layers sequentially across devices, with each GPU hosting a contiguous slice of the network. Activations are passed forward and gradients backward in a micro-batch pipeline.

• Minimizes cross-device communication to layer boundaries
• Susceptible to pipeline bubbles where devices idle waiting for dependencies
• GPipe and PipeDream are seminal scheduling algorithms that reduce idle time
• Ideal for deep, homogeneous architectures like large transformers

Shards individual weight matrices within a single layer across multiple devices, parallelizing the matrix multiplication itself. Each device computes a partial result that is then aggregated.

• Requires high-bandwidth, low-latency interconnects like NVLink or InfiniBand
• Communication happens on every forward and backward pass
• Megatron-LM popularized this for transformer self-attention and MLP blocks
• Enables training and serving of models with trillions of parameters

Distributes the sequence dimension of activations across devices alongside tensor parallelism. This prevents redundant memory consumption from layer normalization and dropout operations.

• Complements tensor parallelism by sharding non-matrix-multiply operations
• Reduces peak activation memory, enabling longer context lengths
• Uses all-gather and reduce-scatter collectives for synchronization
• Critical for long-context inference in models like Llama 3 and GPT-4

Places different mixture-of-experts (MoE) sub-networks on separate devices. Each token is dynamically routed to a subset of experts, and only those experts are activated.

• Dramatically increases total parameter count without proportional compute cost
• Requires all-to-all communication to dispatch and combine tokens across devices
• DeepSpeed-MoE and Switch Transformer are reference implementations
• Enables sparse activation where only 5-10% of parameters are used per token

Combines multiple parallelism strategies to optimize for a specific hardware topology. A common configuration uses tensor parallelism within a node (NVLink) and pipeline parallelism across nodes (InfiniBand).

• 3D parallelism: data + tensor + pipeline parallelism simultaneously
• ZeRO (Zero Redundancy Optimizer) partitions optimizer states, gradients, and parameters
• FSDP (Fully Sharded Data Parallel) shards parameters across all devices
• Balances memory efficiency, communication overhead, and compute utilization

Model parallelism relies on collective communication operations to synchronize partial results. The choice of primitive directly impacts throughput and latency.

• All-reduce: Aggregates gradients or activations across all devices
• All-gather: Each device broadcasts its shard to all others
• Reduce-scatter: Reduces data and scatters results, the inverse of all-gather
• Point-to-point send/recv: Used in pipeline parallelism for passing activations
• NCCL (NVIDIA Collective Communications Library) provides GPU-optimized implementations

A specific form of model parallelism where individual weight matrices or tensors within a single transformer layer are sharded across multiple GPUs. Each device computes a partial result for the same layer, and an all-reduce collective communication operation synchronizes the outputs before proceeding to the next layer. This technique reduces the memory footprint per accelerator and is critical for models where even a single layer's parameters exceed the VRAM of one GPU. Often implemented via libraries like Megatron-LM.

A complementary strategy to tensor parallelism where the model is partitioned by layer depth rather than layer width. Different groups of consecutive layers are assigned to different GPUs. The input data is split into micro-batches that flow through the pipeline, keeping all devices busy. The primary challenge is the pipeline bubble—idle time at the start and end of a batch. Techniques like 1F1B (one-forward-one-backward) scheduling minimize this bubble.

A collective communication primitive fundamental to distributed training and tensor parallelism. Each participating device starts with its own partial data array. The all-reduce operation aggregates these arrays (e.g., via summation) and distributes the final result back to every device. In model parallelism, this synchronizes partial outputs across GPUs. Optimized implementations like NVIDIA NCCL use ring or tree algorithms to maximize inter-GPU bandwidth and minimize synchronization overhead.

The contrasting distributed strategy where a complete copy of the model is placed on each accelerator, but each device processes a different subset (shard) of the training batch. Gradients are synchronized across devices after each backward pass using an all-reduce operation. While simpler than model parallelism, it fails when a single model is too large to fit on one device. Modern systems often combine FSDP (Fully Sharded Data Parallelism) to shard parameters during training, blurring the line with model parallelism.

The combined orchestration of data parallelism, tensor parallelism, and pipeline parallelism to scale training across thousands of GPUs. This is the strategy used to train models like GPT-3 and Megatron-Turing NLG. The system is partitioned into data-parallel groups, within which models are split via tensor and pipeline parallelism. This maximizes throughput while respecting hardware constraints like NVLink bandwidth within a node versus slower inter-node connections.

A memory optimization framework introduced by DeepSpeed that reduces per-device memory consumption without model code changes. ZeRO Stage 3 partitions not just optimizer states and gradients, but also model parameters across data-parallel workers, effectively behaving like model parallelism. Parameters are fetched from other devices on-demand during forward and backward passes via high-speed interconnects. This allows training models with trillions of parameters on commodity hardware clusters.