Early Exit

A backbone network augmented with internal classifiers attached to intermediate layers. Each classifier independently assesses feature representations and can emit a prediction. The architecture is trained end-to-end using a joint loss function that sums weighted losses from all exits, ensuring shallow layers learn discriminative features without degrading the final exit's accuracy.

Inference proceeds layer-by-layer until an exit's prediction confidence exceeds a calibrated threshold. Common metrics include:

• Softmax probability: max predicted probability
• Entropy: normalized entropy of the output distribution
• Agreement: consensus among multiple shallow classifiers This mechanism converts a fixed-depth network into a variable-depth, input-adaptive inference engine.

Shallow exits often underperform when trained solely on task labels. Knowledge distillation uses the final exit's soft logits as a supervisory signal for earlier classifiers. The combined loss function: L_total = Σ w_i * L_CE(y, ŷ_i) + λ * L_KD(ŷ_final, ŷ_i) where w_i weights each exit and λ balances distillation strength. This ensures early exits mimic the full model's behavior.

Early exit networks enable dynamic resource allocation at inference time. Simple or familiar inputs exit at shallow layers using minimal FLOPs, while ambiguous or novel samples propagate deeper. This creates a Pareto-optimal trade-off between accuracy and compute, allowing a single model to serve diverse latency budgets without retraining or maintaining multiple model variants.

On resource-constrained edge hardware, early exit networks provide graceful degradation under load. When compute budgets are tight, the confidence threshold can be lowered to force earlier exits, maintaining throughput at a controlled accuracy cost. This pairs naturally with split computing architectures, where the shallow exits reside on-device and deeper layers execute on an edge server only when necessary.

Early exit networks are a practical realization of anytime inference—the property that a model can be interrupted at any point and still produce a valid result. Unlike traditional anytime algorithms that require explicit intermediate representations, early exits provide native, learned intermediate predictions that improve monotonically with depth, making them ideal for hard real-time systems with strict deadline guarantees.

The parent category of Early Exit mechanisms. This design principle activates or skips portions of a network on a per-input basis using gating networks or confidence thresholds.

• Gating mechanisms learn to route easy samples through shallow paths
• Hard samples still utilize full network depth
• Can reduce FLOPs by 30-50% without proportional accuracy loss
• Includes: early exits, mixture-of-experts, and dynamic layer skipping

A model property where inference can be interrupted at any point and still produce a valid, monotonically improving result. Early Exit is one mechanism to achieve this.

• Guarantees a result within a hard real-time deadline
• Prediction quality improves the longer the model runs
• Critical for safety-critical edge applications with strict latency budgets
• Often combined with anytime prediction heads at multiple depths

A complementary compression technique where a compact student model is trained to replicate a larger teacher. Unlike Early Exit, distillation produces a statically smaller model rather than dynamic depth adjustment.

• Student trained on teacher's soft logits, not just hard labels
• Can be combined with early exit branches in the student
• Produces fixed-size models suitable for microcontrollers
• Often used alongside quantization for extreme compression

The strategic division of a DNN's computational graph for distributed execution across device and edge server. Early Exit branches can serve as natural partition points.

• Head executes on-device up to the exit branch
• If confidence is low, the tail executes on the edge server
• Reduces device compute and energy consumption
• Exit branches at shallow layers minimize transmitted data

Couples predictions with calibrated confidence estimates to enable risk-aware decision making. Early Exit relies on confidence thresholds—uncertainty quantification makes these thresholds statistically rigorous.

• Bayesian methods provide principled uncertainty estimates
• Evidential deep learning predicts uncertainty in a single forward pass
• Prevents overconfident early exits on out-of-distribution inputs
• Essential for safety-critical applications like medical imaging

An adaptive decision engine that determines in real-time whether to execute locally or offload to an edge server. Early Exit confidence scores feed directly into this decision.

• High confidence → return result immediately on-device
• Low confidence → offload deeper computation to MEC server
• Considers network conditions, device battery, and latency budget
• Closely related to QoS-Aware Partitioning strategies