Inference Offloading Decision Engine

The engine continuously evaluates a weighted cost function that balances latency, energy consumption, and inference accuracy. It does not simply minimize one variable; it finds a Pareto-optimal point based on user-defined policies.

• Latency term: Sum of compute time, network transmission, and queuing delay.
• Energy term: Measured in joules for on-device execution vs. transmission power.
• Accuracy penalty: Quantifies degradation from compression or early exit.

A typical formulation: Cost = w₁·Latency + w₂·Energy + w₃·(1 - Accuracy) where weights are dynamically adjusted based on battery level and application criticality.

The engine consumes a stream of channel state information (CSI), available bandwidth, and round-trip time measurements to predict transmission latency before making an offloading decision.

• Ingests PHY-layer metrics like SINR and MCS index from the modem.
• Maintains a sliding window of historical link quality to forecast short-term degradation.
• Uses exponential moving averages to smooth noisy measurements without introducing phase lag.

Without accurate telemetry, the engine defaults to conservative on-device execution, sacrificing potential edge acceleration.

A pre-computed registry stores the computational characteristics of every deployable model variant, enabling the engine to reason about offloading without runtime benchmarking.

• FLOPs per layer: Floating-point operations required for each DNN segment.
• Intermediate tensor size: Byte count of activations at every potential partition point.
• Hardware-specific latency: Profiled execution times on target device SoCs and edge GPUs.

This database is generated offline using ONNX Runtime or Triton Inference Server profiling tools and updated whenever a new model version is registered.

Rather than a binary on-device vs. offload decision, the engine selects an optimal bottleneck layer for split computing. It evaluates every viable cut point in the model's computational graph.

• Head execution: Layers before the partition run on-device, extracting compressed features.
• Tail execution: Remaining layers run on the MEC server, leveraging hardware acceleration.
• Transmission cost: The size of the intermediate feature map determines network payload.

The engine uses a cost-table lookup rather than solving an NP-hard graph partitioning problem at runtime, ensuring decisions complete in microseconds.

The engine integrates with models that support conditional computation through intermediate classifier branches. When a shallow classifier achieves sufficient confidence, execution terminates without invoking deeper layers or offloading.

• Monitors softmax entropy at each exit point as a confidence proxy.
• Compares against a dynamically adjusted confidence threshold tied to the cost of misclassification.
• For latency-critical applications, the threshold relaxes to favor early termination.

This mechanism is particularly effective for easy inputs that dominate real-world distributions, providing sub-millisecond inference for the majority of requests.

The engine explicitly models and controls tail latency (p95, p99) rather than optimizing for mean performance alone. It incorporates server-side queue depth and dynamic batching status into its decision logic.

• If the edge server's inference queue exceeds a threshold, the engine biases toward local execution.
• Uses H2O (Heavy-Hitter Oracle) style prioritization to identify requests likely to cause head-of-line blocking.
• Maintains a deadline budget per request; if offloading cannot meet the SLO, computation stays local.

This ensures consistent user experience under variable load, preventing the long-tail latency spikes that degrade perceived responsiveness.