Model Partitioning

Partitions a model by layer depth, assigning contiguous groups of layers to different devices. Each device processes its assigned layers and passes intermediate activations to the next device in the pipeline.

• Micro-batching splits input data into smaller chunks to keep all devices busy concurrently, reducing idle time between pipeline stages
• Commonly used for large language models and deep vision transformers where layer count exceeds single-GPU memory
• Introduces a bubble overhead at pipeline start and end, which can be minimized with scheduling optimizations like GPipe or PipeDream
• Example: A 96-layer transformer can be split across 4 GPUs, each handling 24 layers sequentially

Splits individual weight matrices within a single layer across multiple devices, parallelizing the matrix multiplication operations themselves. Each device holds a shard of the weight tensor and computes a partial result.

• Requires high-bandwidth interconnects like NVLink or InfiniBand due to frequent all-reduce communication between devices after every forward and backward pass
• Ideal for layers with massive parameter counts that exceed the memory of a single accelerator
• Often combined with pipeline parallelism in a hybrid 3D parallelism strategy for training and inference of trillion-parameter models
• Example: A single attention head's QKV projection matrix can be split column-wise across 4 GPUs, with each computing a portion of the output

Leverages the Mixture of Experts (MoE) architecture, where different sub-networks called experts are placed on separate devices. A gating mechanism routes each input token to only a subset of experts, activating only a fraction of the total parameters per inference.

• Dramatically increases total model capacity without proportionally increasing compute cost, as only top-k experts are activated per token
• Introduces load balancing challenges to prevent expert starvation or overload, requiring auxiliary loss functions during training
• Well-suited for edge deployments where expert shards can be distributed across a cluster of lower-power nodes
• Example: Mixtral 8x7B uses 8 experts with 2 active per token, achieving 47B-parameter quality at 13B-parameter inference cost

Replicates the entire model across multiple devices, with each replica processing a different subset of the input batch. Gradients are synchronized across replicas after each backward pass during training.

• During inference, this pattern enables horizontal scaling to increase throughput by serving multiple inference requests concurrently on separate replicas
• Does not reduce the memory footprint of a single model instance, so it cannot solve the problem of a model being too large for one device
• Often combined with dynamic batching to maximize utilization of each replica's compute capacity
• Example: A quality inspection model replicated across 10 edge nodes, each processing images from a different camera on the production line

Distributes the long sequence dimension of input data across multiple devices, complementing tensor parallelism for transformer models processing extended contexts. Each device computes attention and feed-forward operations on its assigned sequence segment.

• Critical for models handling long context windows exceeding 8K tokens, where activation memory from the attention mechanism dominates total memory consumption
• Uses ring-style communication patterns to share key-value pairs between devices during the attention computation
• Reduces peak activation memory proportionally to the number of devices, enabling inference on sequences that would otherwise cause out-of-memory errors
• Example: A 32K-token context can be split across 4 GPUs, each processing 8K tokens with cross-device attention communication

Splits a model between an edge device and a cloud server, executing initial layers locally for feature extraction and offloading computationally intensive later layers to the cloud. This balances latency, privacy, and compute constraints.

• Early exit branches can be inserted at the partition point, allowing the edge to make fast predictions for simple inputs without invoking the cloud
• Reduces data transmission by sending compact intermediate representations rather than raw sensor data, preserving bandwidth and enhancing privacy
• Requires careful bottleneck design at the partition point to minimize the size of transmitted tensors while preserving sufficient information for accurate downstream processing
• Example: A defect detection model runs the first 3 convolutional layers on a smart camera, sending feature maps to a factory-edge server for final classification layers

A compiler optimization that merges multiple discrete neural network operations into a single kernel launch. When partitioning a model, operator fusion reduces the number of inter-device communication points by consolidating adjacent layers that would otherwise require separate data transfers. This directly lowers memory bandwidth bottlenecks and kernel launch overhead.

• Fuses element-wise operations like ReLU with preceding convolutions
• Reduces the number of tensors that must cross partition boundaries
• Critical for maintaining throughput when layers are split across NPU and CPU

A system architecture that combines different types of processing units—CPUs, GPUs, FPGAs, and NPUs—to execute workloads on the most efficient silicon for each task. Model partitioning is the software strategy that maps specific neural network layers to the optimal processor.

• Convolutional layers often target NPUs for energy efficiency
• Transformer attention heads may run on GPUs for parallel throughput
• Lightweight post-processing executes on the CPU to minimize data movement

A software intermediary that decouples AI model code from underlying hardware specifics. When partitioning occurs, the HAL provides a unified interface so that different partitions can target different backends without rewriting application logic.

• Enables a single model to execute across diverse chipsets
• Abstracts memory allocation and synchronization primitives
• Simplifies shadow mode deployment of partitioned models across heterogeneous fleets

An architectural pattern where each AI model partition is packaged as a lightweight, isolated container with its own dependencies. This enables independent scaling and versioning of model segments across an edge cluster managed by orchestrators like K3s.

• Each partition runs in its own container with dedicated resource limits
• Inter-container communication uses gRPC or shared memory for low latency
• Enables rolling updates of individual partitions without full model redeployment

A guaranteed maximum time window within which a computation or data transfer will complete. When partitioning a model across edge nodes connected via Time-Sensitive Networking (TSN), deterministic latency ensures that tensor exchanges between partitions meet the deadlines required for closed-loop industrial control.

• Non-negotiable for safety-critical manufacturing applications
• Requires bounded jitter on inter-partition communication links
• Often paired with Real-Time Operating Systems (RTOS) on each edge node