Pipeline Parallelism

The neural network is split into sequential stages, each containing one or more contiguous layers. Stage 0 might hold the embedding and first transformer block, while Stage N holds the final classification head. Each stage is assigned to a dedicated accelerator, and activations flow forward while gradients flow backward during training. This partitioning is typically done manually based on layer computational cost and memory footprint to balance load across devices.

Instead of processing a single batch end-to-end, the input batch is divided into smaller micro-batches. These micro-batches are injected into the pipeline in a staggered fashion, allowing multiple stages to compute simultaneously on different micro-batches. Common scheduling algorithms include:

• GPipe: Flushes the pipeline after each forward pass before starting backward
• 1F1B (One-Forward-One-Backward): Interleaves forward and backward passes to reduce memory footprint
• Interleaved 1F1B: Further subdivides stages for better load balancing

To reduce peak memory consumption, pipeline parallelism often employs activation checkpointing. Instead of storing all intermediate activations for the backward pass, only the inputs to each pipeline stage are retained. During backpropagation, forward activations are recomputed on-demand from these checkpoints. This trades a modest increase in computation (typically ~33%) for a significant reduction in memory, enabling the training of models that would otherwise exceed device capacity.

The pipeline bubble refers to idle time where some accelerators wait for data from upstream stages. The bubble size is proportional to (P - 1) / M, where P is the number of pipeline stages and M is the number of micro-batches. Increasing the number of micro-batches reduces the relative bubble overhead, but increases memory consumption. Advanced scheduling like Chimera uses bidirectional pipelines to eliminate bubbles entirely by having each device participate in two pipeline stages.

Pipeline parallelism is rarely used in isolation. It is combined with data parallelism (replicating the pipeline across multiple groups) and tensor parallelism (splitting individual layers across devices) to form 3D parallelism. In this hybrid approach, the model is first partitioned by layer into pipeline stages, then each large layer is sharded via tensor parallelism, and finally the entire configuration is replicated for data parallelism. This is the standard strategy for training models exceeding 100 billion parameters.

Unlike tensor parallelism which requires all-reduce collectives, pipeline parallelism uses simple point-to-point send/recv operations between adjacent stages. Activations flow from Stage i to Stage i+1, and gradients flow in reverse. This communication pattern is bandwidth-efficient and maps well to high-speed interconnects like NVLink and InfiniBand. The sequential dependency means that communication latency is hidden by overlapping computation with transfer using CUDA streams or asynchronous copy engines.

In high-bandwidth direct RF sampling architectures, a single FPGA often lacks the resources to run a complete deep learning model. Pipeline parallelism enables a digital down converter, cyclic prefix removal, and initial convolutional layers to execute on a first FPGA, while subsequent transformer encoder blocks run on a second device connected via AXI4-Stream or JESD204B links.

• Benefit: Maintains real-time throughput on streaming IQ samples exceeding 1 GSPS.
• Mechanism: Each FPGA processes its assigned sub-graph and forwards intermediate feature maps, not raw samples, to the next stage.

Microcontrollers with integrated Neural Processing Units (NPUs) often have limited on-chip memory, preventing the loading of a full model. Pipeline parallelism splits the workload: the Cortex-M CPU handles lightweight signal preprocessing and feature extraction, while the Ethos-U NPU executes the computationally intensive classification head.

• Example: A keyword spotting model where the CPU computes Mel-frequency cepstral coefficients (MFCCs) and the NPU runs a depthwise separable convolutional network.
• Result: Achieves sub-10ms latency on battery-powered devices by keeping both compute units active simultaneously.

A common edge robotics pattern uses pipeline parallelism to decouple region proposal from classification. A lightweight backbone (e.g., MobileNet) runs on the Jetson's GPU to identify regions of interest, while a heavier Vision Transformer on a connected desktop GPU refines the bounding boxes.

• Data Flow: The edge device transmits only cropped feature tensors, not full-resolution video frames, over a wired Ethernet link.
• Optimization: This minimizes the memory bandwidth bottleneck on the embedded GPU, allowing it to process higher camera frame rates.

In a multi-camera tracking system, pipeline parallelism distributes model stages across a hierarchy of devices. Edge cameras perform person detection and feature embedding extraction locally. A central edge server aggregates embeddings from all streams and runs a re-identification (Re-ID) graph neural network.

• Topology: A logical pipeline where the initial layers are massively parallelized across cameras, and the final matching layers are centralized.
• Latency Profile: End-to-end latency is bounded by the slowest stage, typically the centralized Re-ID network, which can be accelerated with TensorRT.

For privacy-sensitive applications, pipeline parallelism enables split computing, where a model is physically partitioned between an untrusted edge device and a trusted on-premise server. The edge device executes initial layers on raw data, then transmits an obfuscated bottleneck feature vector to the server for final classification.

• Security: Raw sensor data never leaves the edge device.
• Use Case: Medical imaging analysis where a hospital's edge scanner runs the encoder and a secure internal server runs the diagnostic decoder.

An SDR pipeline for automatic modulation classification and RF fingerprinting benefits from pipeline parallelism by separating signal conditioning from neural network inference. A Xilinx Zynq's programmable logic handles IQ imbalance correction and time-frequency transforms, while its ARM cores run a TinyML classifier on the extracted features.

• Throughput: The FPGA fabric streams processed spectrograms directly to the classifier via zero-copy transfer, avoiding CPU bottlenecks.
• Outcome: Enables real-time emitter identification on a single heterogeneous SoC.