Conditional Computation

Gating networks are small, learned sub-modules that make discrete routing decisions for each input sample. They analyze the input representation and selectively activate only the most relevant expert sub-networks or layers.

• Hard Gating: Makes a binary, non-differentiable choice, routing the input to a single expert (e.g., Mixture-of-Experts).
• Soft Gating: Computes a weighted average of all expert outputs, allowing gradients to flow through all paths during training.
• Top-k Gating: A sparse variant that activates only the k experts with the highest gating scores, balancing efficiency and model capacity.

A prominent conditional computation architecture where a large model is composed of many smaller, specialized expert networks. For each input token, a gating network activates only a sparse subset of these experts.

• Sparsity: A key property where only a fraction of total parameters are used per inference, drastically reducing FLOPs.
• Load Balancing: A critical auxiliary loss is often added to prevent the gating network from collapsing and always routing to the same few experts.
• Scaling Law: MoE enables training models with trillions of parameters while keeping inference compute sub-linear, as seen in Mixtral 8x7B.

A conditional computation strategy that attaches intermediate classifier branches to the shallow layers of a deep network. If a prediction confidence threshold is met at an early exit, computation halts immediately.

• Adaptive Computation Time: The model dynamically allocates more processing layers only for difficult or ambiguous inputs.
• Confidence Thresholding: A hyperparameter controls the trade-off between speed and accuracy; a higher threshold forces more samples through deeper layers.
• Latency Reduction: Simple inputs can be classified in a fraction of the time, making this ideal for real-time edge inference with variable input complexity.

Techniques that conditionally discard redundant information within a single forward pass to reduce the quadratic complexity of attention mechanisms.

• Token Merging: Dynamically combines similar, redundant tokens (e.g., background patches in an image) into a single representative token to reduce sequence length.
• Token Pruning: Selectively removes tokens deemed unimportant by an attention-based scoring function, preventing them from being processed by subsequent transformer layers.
• Layer Dropping: Stochastically or deterministically skips entire residual blocks during training and inference, creating a dynamic network depth.

Conditional computation can be applied to the structural dimensions of a network, not just its expert selection. The architecture itself adapts per input.

• Dynamic Depth: A halting score determines how many sequential layers an input should traverse before exiting, as seen in Adaptive Computation Time (ACT) for RNNs.
• Dynamic Width: A gating signal can selectively prune entire channels or filters in a convolutional layer, effectively running a thinner model for easy inputs.
• Compute Budgeting: These methods can be tuned to enforce a strict average FLOPs budget, guaranteeing a target throughput while maximizing accuracy.

Instead of conditioning which layers execute, this approach conditions how layers behave by modulating normalization parameters based on auxiliary information.

• Feature-wise Linear Modulation (FiLM): Applies a learned, input-dependent affine transformation to a layer's feature maps, effectively gating and shifting information flow.
• Conditional Batch Normalization: Uses a separate set of learned scale and shift parameters for different classes or styles, allowing a single generator to produce diverse outputs.
• Spatially-Adaptive Normalization (SPADE): A more granular form where the modulation parameters are spatially varying, enabling precise control over semantic layout in image synthesis.

A lightweight, learned routing function that makes the discrete decision about which expert or sub-network to activate for a given input. The gating network typically outputs a sparse probability distribution over experts, often using a top-k softmax to enforce hard sparsity. The key challenge is training this non-differentiable, discrete decision process, usually solved via gradient estimation techniques like REINFORCE or straight-through Gumbel-Softmax. Without careful regularization, gating networks are prone to mode collapse, where they converge to always selecting the same few experts.

The foundational conditional computation architecture where a model comprises multiple expert sub-networks and a gating mechanism. For each input token or sample, only a sparse subset of experts is activated. This enables a dramatic increase in total parameter count without a proportional increase in FLOPs per inference. Key variants include:

• Sparse MoE: Only the top-k experts are activated per input
• Switch Transformer: Simplifies to top-1 routing, reducing communication overhead
• Expert Choice Routing: Experts choose the top-k tokens, ensuring perfect load balance

An auxiliary loss function added during training to prevent the gating network from collapsing to a few over-specialized experts. Without it, early-favored experts receive more gradients and dominate, starving others. The loss encourages a uniform distribution of tokens across experts. Common formulations include:

• Auxiliary loss: Penalizes deviation from a balanced expert assignment distribution
• z-loss: Penalizes large logits entering the router to stabilize training
• Expert capacity factor: Hard limit on tokens per expert, with overflow tokens dropped or passed through residual connections

A conditional computation strategy where internal classifiers are attached to intermediate layers of a deep network. If a sample's prediction confidence exceeds a predefined threshold at an early exit, computation halts and the result is returned immediately. This creates a dynamic depth network where 'easy' inputs use fewer layers. Key design decisions include:

• Confidence threshold calibration: Balancing speed vs. accuracy trade-off
• Exit placement: Where to attach classifiers for optimal cost reduction
• Joint training: Training all exits simultaneously with a weighted sum of losses

A mechanism where the scale and shift parameters of batch normalization layers are dynamically generated based on a conditioning signal, rather than being static learned parameters. This allows a single backbone network to adapt its feature normalization to different computation paths or input types. It is a lightweight form of conditional computation that modulates behavior without changing the network's structural connectivity. Commonly used in visual reasoning and style transfer where the conditioning signal encodes task identity or stylistic parameters.

Unlike static pruning that permanently removes weights, dynamic pruning makes input-dependent sparsity decisions at runtime. For each input, a subset of weights, neurons, or channels is zeroed out based on a learned importance predictor. This is distinct from MoE because it operates on continuous weight magnitudes rather than discrete expert selection. Techniques include:

• Gating modules: Small networks that predict which channels to skip
• Threshold-based activation: Only neurons with activations above a learned threshold propagate
• Runtime neural architecture search: Selecting sub-networks on-the-fly