Adaptive Computation

Early exiting allows a neural network to produce a prediction from an intermediate layer if the input is deemed sufficiently simple, bypassing deeper, more computationally expensive layers. This is governed by internal confidence thresholds or routing classifiers.

• Mechanism: A small classifier or gating function is attached to intermediate layers. If its confidence score exceeds a predefined threshold, the inference halts, and the current layer's output is used as the final prediction.
• Benefit: Dramatically reduces average inference latency and compute cost for 'easy' samples, such as classifying clear images or simple sentences.
• Example: The PABEE (Patience-based Early Exit) framework for BERT models allows tokens to exit at different layers based on stability of predictions.

Conditional computation dynamically activates only a subset of a model's parameters for a given input. The most prominent architecture is the Mixture of Experts (MoE), where a router network selects a small number of specialized 'expert' sub-networks to process each token.

• Mechanism: A gating network computes a sparse combination of experts. Only the weights of the selected experts are used in the forward pass.
• Benefit: Enables massive model scale (e.g., trillion-parameter models) without a proportional increase in computational cost per token, as computation scales with the number of active experts, not total parameters.
• Example: Switch Transformers use a simplified MoE layer where the router selects a single expert per token, achieving high efficiency.

Adaptive attention mechanisms modify the standard self-attention operation in Transformers to focus computation on the most relevant token interactions, reducing the quadratic O(n²) cost.

• Key Techniques:
  • Sparse Attention: Uses a fixed, predefined pattern (e.g., local windows, strided patterns) to limit the number of token pairs each token attends to.
  • Adaptive Sparse Attention: Dynamically selects which tokens to attend to based on content, using learned sparsity patterns or locality-sensitive hashing (LSH).
  • Recycled Attention: Reuses attention scores from previous layers or computation steps.
• Benefit: Enables processing of extremely long sequences (documents, books) that are infeasible with standard dense attention.

Dynamic neural networks encompass architectures where the computational graph itself—the path data takes through the network—adapts per input. This goes beyond early exiting to include dynamic depth, width, and feature transformation.

• Dynamic Depth: The number of layers executed varies per sample (as in early exiting).
• Dynamic Width: The number of channels or neurons activated within a layer varies. Channel gating techniques learn to shut off unimportant channels.
• Dynamic Feature Transformation: The type of operation applied (e.g., convolution kernel size) is input-dependent.
• Benefit: Provides fine-grained, per-layer control over computational cost, allowing the model to allocate more resources to ambiguous or complex features within a single input.

Input-adaptive pruning (or dynamic sparsity) removes different sets of weights from the network for different inputs, in contrast to static pruning which removes the same weights for all inputs.

• Mechanism: A lightweight auxiliary network or gating function predicts a binary mask that zeroes out a subset of weights in the main network for the current input. The sparsity pattern is therefore sample-specific.
• Benefit: Achieves higher sparsity rates (and thus greater compute reduction) than static pruning for a given accuracy budget, as the model can retain a diverse set of weights across different inputs.
• Challenge: Requires efficient hardware support for irregular, dynamic sparse matrix operations to realize speedups.

Confidence-based cascades arrange multiple models of increasing size and capability in a sequence. A smaller, faster model processes the input first; only if its prediction confidence is below a threshold is the input passed to the next, larger model.

• Mechanism: Implements a model cascade or hierarchy. This is a system-level application of early exiting, often using entirely distinct model architectures (e.g., DistilBERT -> BERT -> BERT-Large).
• Benefit: Maximizes system-wide throughput and minimizes average latency by ensuring the vast majority of 'easy' queries are answered by the smallest, cheapest model. It is a foundational technique for cost-effective AI API services.
• Example: Google's Cascading Models for visual recognition tasks use this principle to achieve low latency at scale.

A conditional computation architecture where a routing network dynamically selects a sparse subset of specialized 'expert' sub-networks to process each input. This enables massive model capacity (e.g., trillions of parameters) without a proportional increase in computation per token, as only a few experts are activated.

• Key Mechanism: A gating network computes weights for each expert; only the top-k experts are activated.
• Example: Models like Switch Transformers and Mixtral 8x7B use MoE to achieve state-of-the-art performance with manageable inference costs.
• Trade-off: Increases parameter count but decreases active FLOPs, requiring sophisticated distributed systems for training and serving.

An adaptive inference technique where a neural network contains multiple exit points (classifiers) at intermediate layers. Simpler inputs that achieve high confidence at an early layer can 'exit' the network, bypassing deeper, more expensive computations.

• Goal: Reduce average inference latency by applying variable-depth processing.
• Architecture: Often implemented in transformer models with classifiers attached to different encoder blocks.
• Use Case: Ideal for processing workloads with mixed complexity, such as a stream of user queries where some are straightforward and others are complex.

A broad paradigm where the computational graph or pathway is dynamically determined based on the input. It is the super-category that includes techniques like Mixture of Experts and Early Exiting.

• Core Principle: The model's architecture is not static; it's a function of the input data.
• Benefits: Enables more flexible and efficient use of computational resources compared to fixed, dense networks.
• Challenges: Introduces dynamic, data-dependent control flow, which can complicate batching, hardware optimization, and reproducibility.

Techniques that train or run neural networks with inherently sparse connectivity, meaning many weights are permanently zero. This reduces memory footprint and FLOPs.

• Structured Sparsity: Pruning weights in hardware-friendly patterns (e.g., 2:4 sparsity, where 2 of every 4 weights are non-zero) to leverage NVIDIA's Sparse Tensor Cores.
• Sparse Training: Training a network from scratch with a fixed sparse mask, avoiding the pre-train/prune/fine-tune cycle.
• Relation to Adaptive Computation: While often static per model, sparsity can be combined with adaptive techniques (e.g., dynamically sparse attention in Sparse Transformers).

A suite of techniques to reduce a model's memory footprint and computational requirements for deployment, often serving as a prerequisite or complementary approach to adaptive computation.

• Key Techniques: Pruning (removing unimportant weights), Quantization (reducing numerical precision of weights/activations), and Knowledge Distillation (training a small model to mimic a large one).
• Difference from Adaptive Computation: Compression typically creates a single, smaller static model. Adaptive computation creates a single model that uses variable compute per input.
• Synergy: A compressed model (e.g., via quantization) can be used as the base for an adaptive system to achieve even greater efficiency.

The overarching field of neural network architectures whose structure or parameters can change during inference based on the input. This is the academic umbrella term for adaptive computation.

• Taxonomy: Includes spatial adaptation (e.g., skipping layers, conditional execution) and temporal adaptation (e.g., adaptive computation time in RNNs).
• Motivation: Mirrors the efficiency of biological brains, which allocate more resources to difficult stimuli.
• Engineering Impact: Pushes the limits of traditional deep learning frameworks, requiring support for dynamic control flow and specialized kernels.