Model Pipelining

Model pipelining achieves parallelism by dividing a sequential workload—such as a Retrieval-Augmented Generation (RAG) pipeline with distinct retriever, reranker, and generator stages—into independent stages that operate concurrently. While Stage 2 processes the output from Stage 1, Stage 1 can begin work on the next data sample (e.g., a new user query). This overlapped execution converts a purely sequential latency into a higher, sustained throughput, which is critical for handling multiple concurrent requests on edge servers or gateways.

• Example: An edge RAG pipeline where the embedding model retrieves documents for Query N+1 while the small language model generates an answer for Query N.

A core feature is the intelligent mapping of pipeline stages to the most suitable available hardware. Different stages have heterogeneous compute requirements. For instance, a dense retriever performing vector search is memory-bandwidth bound, while an LLM generator is compute-intensive. Pipelining allows the system to assign:

• The retrieval stage to a CPU with high memory bandwidth.
• The reranking stage to an integrated GPU.
• The generation stage to a dedicated Neural Processing Unit (NPU). This maximizes the utilization of all available silicon on a system-on-chip (SoC), preventing any single resource from becoming the bottleneck and improving overall energy efficiency.

To keep all pipeline stages continuously busy and hide communication latency between devices, pipelining operates on micro-batches of data rather than single samples. A micro-batch is a small group of inputs (e.g., 2-4 queries) that flows through the pipeline as a unit. This increases arithmetic intensity and improves hardware utilization, especially for accelerators like GPUs/NPUs that perform best with batched operations. The size of the micro-batch is a tunable parameter that balances latency and throughput, often adjusted dynamically based on current load and available memory.

Efficient pipelining requires managed buffers between stages to hold intermediate results. These buffers decouple the execution rates of adjacent stages. Key design considerations include:

• Fixed-size vs. Dynamic Queues: Managing backpressure when a downstream stage is slower.
• Memory Location: Using shared system RAM, GPU memory, or fast on-chip SRAM depending on the connected hardware stages.
• Data Format: Often using efficient serialization formats like Protocol Buffers or raw tensors to minimize serialization overhead. Poor buffer management can lead to stalls, defeating the purpose of pipelining.

Pipelining is particularly effective at hiding the latency of I/O-bound stages. In an edge RAG system, the retrieval stage may involve fetching data from a local vector database or solid-state drive, which has high latency compared to compute. By having the subsequent reranking and generation stages process previous queries while the retriever fetches data for the next query, the long I/O latency is overlapped with useful computation. This transforms what would be additive, sequential wait times into a less impactful component of the overall pipeline latency.

Advanced pipelining systems can reconfigure the pipeline graph at runtime based on system state. This is crucial for edge environments where resource availability fluctuates. Examples include:

• Stage Bypassing: Skipping a non-essential reranker stage under high load to reduce latency.
• Compute Offloading: Dynamically moving a computationally heavy stage (like the generator) to a neighboring edge server if the local NPU becomes overheated or busy, while keeping lighter stages local.
• Alternative Model Selection: Switching to a more lightweight, quantized model for a specific stage to maintain throughput during thermal throttling.

A dynamic resource management strategy within an edge RAG pipeline. It involves selectively executing computationally intensive components (e.g., the large LLM generator) on a neighboring server, edge cloudlet, or the cloud, while keeping lighter components (e.g., the retriever, semantic cache) on the local device.

• Balances latency, privacy, and resource constraints.
• The pipeline orchestrator makes offloading decisions based on current device load, network availability, and task criticality.

A minimal-footprint software component that manages the execution flow and data handoff between stages in an edge RAG pipeline. Its responsibilities include:

• Scheduling the concurrent execution of retriever, reranker, and generator stages.
• Managing intermediate data buffers between pipeline stages.
• Implementing dynamic batching and handling compute offloading decisions.
• Providing resilience and fallback mechanisms if a pipeline stage fails.

The optimization of the embedding generation and similarity search components to leverage a device's Neural Processing Unit. In a pipelined system, this typically accelerates the initial retriever stage.

• NPUs excel at the small, parallel matrix multiplications required for transformer-based query/document encoders.
• By offloading this work from the CPU/GPU, it frees resources for concurrent execution of other pipeline stages (e.g., reranking).
• Requires models compiled to specific NPU instruction sets (e.g., via TensorRT-LLM, Qualcomm AI Engine).