Split Computing

The strategic slicing of a deep neural network's computational graph at a specific bottleneck layer. The initial layers (head) execute on the device, processing raw input into a compact, intermediate feature representation. This compressed tensor is transmitted to the edge server, where the remaining layers (tail) complete the inference. The bottleneck is chosen to minimize the transmission overhead while preserving sufficient information for the server-side layers to achieve target accuracy. This contrasts with simple device-cloud offloading by transmitting features, not raw data.

A real-time scheduler that determines the optimal execution target for each inference request. It continuously monitors a vector of telemetry data:

• Device state: CPU load, memory pressure, battery level, thermal headroom.
• Network state: Current bandwidth, latency, jitter, and predicted link quality.
• Model characteristics: Size of the head model, compressed feature size, and tail model complexity. The engine uses a heuristic or lightweight ML model to decide whether to execute locally, split at a specific point, or offload entirely, adapting to fluctuating conditions to meet a strict latency budget.

Techniques applied to the activation tensor at the partition point to drastically reduce bandwidth consumption before transmission. Common methods include:

• Quantization: Reducing the numerical precision of activations from FP32 to INT8 or even lower bit-widths.
• Entropy coding: Applying lossless compression like Huffman or arithmetic coding to the quantized tensor.
• Dimensionality reduction: Using a 1x1 convolutional bottleneck or PCA to shrink the channel dimension. This compression is critical for operating on bandwidth-constrained or metered wireless links without negating the latency benefits of offloading.

An advanced offloading strategy that couples the model's partition point directly to the physical layer conditions. Instead of a static split, the system dynamically selects the optimal bottleneck layer based on real-time Channel State Information (CSI) . Under poor channel conditions (low SNR), the split occurs at a deeper, more compressive layer to minimize payload size. When the link is strong, an earlier split transmits richer features for higher server-side accuracy. This creates a closed-loop system that jointly optimizes for wireless physics and inference accuracy.

A fundamental security property of split computing architecture. By transmitting only the intermediate feature tensor—a highly abstract, non-human-interpretable representation—rather than raw sensor data, the system inherently provides a layer of privacy. The edge server never has direct access to the original image, audio, or video stream. This is often combined with Split Federated Learning, where the device-side head model is trained locally, and only smashed data or gradients are shared, ensuring raw data never leaves the device.

The server-side optimization of the tail model's execution to meet strict Quality of Service guarantees. The edge server employs techniques like dynamic batching to group asynchronous inference requests from multiple devices, maximizing GPU utilization without violating individual latency budgets. Combined with Multi-Instance GPU (MIG) partitioning, the server can provide hardware-level isolation and guaranteed throughput for different service classes, ensuring deterministic tail latency even under highly variable load from a heterogeneous device fleet.

The process of identifying the optimal split point within a neural network. An ideal bottleneck layer minimizes the size of transmitted feature maps while preserving sufficient information for the server-side tail to achieve target accuracy.

• Information bottleneck theory: Balances compression against predictive power
• Empirical profiling: Measuring per-layer output size and inference time on target hardware
• Neural architecture search: Automating the discovery of partition-friendly architectures with natural compression points
• Common candidates include the output of SqueezeNet fire modules or MobileNet inverted residuals

A seamless, multi-tier computing architecture enabling dynamic workload migration. Inference tasks flow between on-device processors, MEC servers at the network edge, and centralized cloud data centers based on latency requirements and resource availability.

• Tier 1 (Device): Sub-millisecond inference for safety-critical functions
• Tier 2 (Edge): 1-10ms inference for interactive applications
• Tier 3 (Cloud): >50ms inference for complex, batch-oriented processing
• Split computing enables fluid transitions between tiers without restarting the inference pipeline