QoS-Aware Partitioning

The partition point is selected to guarantee that end-to-end inference latency remains below a strict Service Level Objective (SLO). The decision engine models the sum of on-device execution time, network transmission latency, and edge server compute time to find a split that satisfies the deadline. For example, a complex object detection model might execute the first 12 layers on a smartphone and offload the remaining 38 layers to an edge server only when the predicted Round-Trip Time (RTT) is under 5 ms. If the channel degrades, the system falls back to a shallower split or full on-device execution to avoid violating the latency budget.

Deeper layers in a neural network extract more abstract, semantically rich features. Partitioning too early forces the edge server to reconstruct the prediction from low-level features, potentially degrading accuracy. A QoS-aware partitioner evaluates the accuracy-latency Pareto frontier for each candidate split point. For instance, splitting a ResNet-50 at layer 3 may yield 20 ms latency but 72% top-1 accuracy, while splitting at layer 10 yields 45 ms latency but 91% accuracy. The system dynamically selects the deepest split that still meets the latency SLO, maximizing accuracy under the given constraints.

The intermediate activations transmitted at the partition point can be substantial—often megabytes per frame. QoS-aware systems apply lossy compression to these feature tensors, with the compression ratio dynamically tuned to available bandwidth. Techniques include:

• Quantization: Reducing activation precision from FP32 to INT8 or INT4.
• Entropy coding: Applying Huffman or arithmetic coding to the quantized tensor.
• Dimensionality reduction: Using a 1x1 convolutional bottleneck to shrink channel depth before transmission. The compression level is selected to minimize distortion while keeping transmission time within the latency budget.

Wireless channel conditions fluctuate on millisecond timescales due to fading, interference, and mobility. A QoS-aware partitioner ingests real-time Channel State Information (CSI)—including Signal-to-Noise Ratio (SNR) and predicted throughput—from the modem. When the scheduler detects a deep fade, it proactively shifts the partition point deeper into the device to reduce payload size or switches to full on-device inference. This closed-loop adaptation prevents bufferbloat and timeout-induced failures that static partitioning schemes cannot avoid.

The core of QoS-aware partitioning is a decision engine that solves a constrained optimization problem per inference request. Inputs include:

• Device state: CPU/GPU utilization, memory pressure, battery level, thermal headroom.
• Network state: Estimated bandwidth, RTT, packet loss rate.
• Model profile: Pre-characterized latency and accuracy for each candidate split point.
• Request priority: Criticality of the inference (e.g., safety-critical vs. best-effort). The engine uses a heuristic solver or a lightweight reinforcement learning policy to select the partition point that maximizes a weighted utility function of accuracy, latency, and energy.

When device or edge resources are saturated, the system must degrade gracefully rather than fail. QoS-aware partitioning implements a fallback hierarchy:

• Primary path: Optimal split point with edge offload.
• Fallback 1: Shallower split with higher compression.
• Fallback 2: Full on-device execution using a smaller distilled model.
• Fallback 3: Return a cached or default prediction with an uncertainty flag. This tiered approach ensures the application remains functional even under severe resource contention, a critical requirement for autonomous systems and industrial automation.