Early Exit Networks

The core design involves inserting branch classifiers or exit points at intermediate layers of a backbone network (e.g., ResNet, MobileNet). Each exit consists of a lightweight classification head attached to the feature maps of that layer. During inference, a confidence threshold (e.g., based on entropy or max softmax probability) is evaluated at each exit. If the confidence exceeds the threshold, inference halts, and that exit's prediction is returned, preventing execution of deeper, more power-intensive layers.

The decision to exit is governed by a gating function that evaluates prediction confidence at each branch. Common metrics include:

• Entropy: Lower entropy indicates higher confidence.
• Maximum Softmax Probability: A higher max probability suggests certainty.
• Margin: Difference between the top two predicted class probabilities. A pre-defined or adaptive threshold determines if the input is 'easy' enough for the current stage. This creates a dynamic computational graph where the inference path length varies per sample, directly translating to variable energy consumption.

All exit classifiers and the backbone are trained end-to-end with a composite loss function, typically a weighted sum of losses from each exit. A common formulation is: Total Loss = Σ (α_i * Loss_i), where Loss_i is the cross-entropy loss at exit i. This ensures early layers learn generally useful features, while later layers refine them for harder cases. Training must balance accuracy at all exits to prevent conflicting gradients or the collapse of early exits.

Early exit networks are built upon standard feature extractor backbones. The choice of backbone (e.g., EfficientNet, ConvNeXt) dictates the base feature quality. Exits are placed after stages where feature maps have sufficient semantic richness for classification. A key efficiency gain is feature reuse; the computationally expensive feature extraction up to an exit is shared across all deeper potential exits, avoiding redundant computation. The architecture is distinct from cascades, which typically use independent models.

The primary tunable parameter is the confidence threshold. A higher threshold forces more samples to deeper exits, increasing both accuracy and energy use. A lower threshold allows more early exits, saving energy but risking accuracy on ambiguous inputs. This creates a Pareto frontier of operating points. Systems can dynamically adjust this threshold based on available battery or latency requirements, enabling runtime optimization for power-aware TinyML deployments.

This architecture is ideal for always-on sensing and class-imbalanced datasets common in IoT. Examples:

• Audio Event Detection: A 'silence' or 'background noise' class can exit at the first layer, while complex speech commands proceed.
• Visual Wake Words: 'No person' exits early; 'Person detected' uses full network.
• Industrial Anomaly Detection: 'Normal' sensor readings exit early; potential anomalies trigger deeper analysis. This aligns with duty cycling principles, minimizing active compute time.

A hardware-level power management technique that dynamically adjusts a processor's operating voltage and clock frequency based on real-time computational workload. For early exit networks, DVFS can be applied aggressively to earlier, simpler classifier layers, saving significant dynamic power.

• Key Mechanism: Lower voltage/frequency for less intensive compute stages.
• TinyML Application: Coupled with early exits to run initial network layers at minimal power states.

A design paradigm that intentionally trades off computational precision or correctness for gains in energy efficiency, performance, or silicon area. Early exit networks are a form of algorithmic approximation, where 'easy' inputs are processed with a less accurate (but cheaper) intermediate classifier.

• Related Techniques: Use of reduced numerical precision (e.g., INT8), pruning, and stochastic computing.
• Energy-Accuracy Trade-off: Directly managed by the confidence thresholds set at each exit point in the network.

The primary benchmark metric for evaluating the energy efficiency of machine learning accelerators and TinyML systems. It measures the number of neural network inferences performed per joule of energy consumed. Early exit networks directly optimize this metric by avoiding full-network inference for a subset of inputs.

• Calculation: Total Inferences / Total Energy (Joules).
• System-Level Impact: Early exits improve system-level inference-per-watt by reducing average energy per inference.

A sensor power management technique where the frequency of data acquisition is dynamically adjusted based on context or detected activity level. This reduces energy consumption during periods of low information content. Early exit networks enable adaptive processing rates, creating a complementary power-saving stack.

• Synergy: Low sensor sampling rate can feed into early-exit networks for ultra-low-power monitoring.
• Use Case: Always-on audio or vibration sensing, where only 'interesting' events trigger deep network analysis.

The fundamental design space in embedded ML where reducing computational cost (energy) typically reduces model prediction accuracy. Early exit networks explicitly parameterize this trade-off via the confidence thresholds at each exit. Engineers can tune these thresholds for a target operational point on the Pareto frontier.

• Design Knobs: Exit confidence thresholds, number of intermediate classifiers, architecture of exit branches.
• Evaluation: Characterized by plotting accuracy vs. average FLOPs or energy per inference.

A canonical early exit network architecture and one of the first to popularize the concept for convolutional neural networks. Introduced by Teerapittayanon et al., BranchyNet incorporates multiple side branches with classifiers attached to intermediate layers, allowing samples to exit early.

• Architectural Contribution: Demonstrated formal training procedures for multi-exit networks.
• Influence: Serves as a foundational reference for most subsequent research in adaptive inference and early exiting.