Mixture-of-Experts (MoE)

The core efficiency mechanism of MoE. For each input token, a gating network selects only the top-k most relevant experts (typically 1 or 2) for activation. This implements conditional computation, where the vast majority of the model's parameters remain inactive for any given input, allowing for models with trillions of parameters while keeping computational cost per token similar to a much smaller dense model.

Expert layers are the specialized sub-networks, usually replacing the standard feed-forward network (FFN) blocks in a transformer. Each expert develops a specialized skill or domain knowledge (e.g., syntax, mathematics, code). In models like Google's Switch Transformer or Mixtral 8x7B, these are independent FFNs. The model's overall capability emerges from the collective knowledge of hundreds or thousands of these experts.

A small, trainable network that decides expert allocation. It takes the token's hidden state as input and outputs a probability distribution over all experts.

• Top-k Routing: Selects the k experts with the highest scores.
• Load Balancing Loss: A critical auxiliary loss term added during training to prevent expert collapse, where the router favors a small subset of experts. This ensures all experts receive sufficient training data.
• Noise for Exploration: Noise is often added to router logits during training to encourage exploration across experts.

A major engineering challenge in MoE systems. Uneven routing can cause:

• Straggler Effects: Slowdowns if one expert receives disproportionate traffic.
• Memory Explosion: All experts must be loaded in memory, even if sparse in computation. Solutions include:
• Expert Capacity: A fixed limit on tokens per expert per batch.
• Auxiliary Balancing Losses (as used in GShard, Switch Transformer).
• Smart GPU/TPU scheduling to handle dynamic, sparse computation graphs.

A technique to create an MoE model from a dense pre-trained checkpoint. Instead of training experts from scratch, the dense model's existing FFN weights are strategically duplicated and diversified. For example, a dense 7B parameter model's FFN layers can be copied to create 8 experts, which are then fine-tuned with routing, effectively 'upcycling' it into a 47B parameter MoE model (like Mixtral 8x7B) with minimal additional pre-training cost.

MoE shares philosophical ground with PEFT through modularity and sparse updates.

• Task-Specific Routing: An MoE model can be adapted to a new task by fine-tuning only the gating network to route to relevant experts, a highly parameter-efficient approach.
• Expert as Adapter: Individual experts can be viewed as large, specialized adapter modules. The gating network performs dynamic adapter selection.
• Combination with PEFT: Techniques like LoRA can be applied within each expert for further efficient adaptation, creating a hierarchical efficient tuning system.

A neural network design principle where different computational pathways are dynamically activated based on the input. This enables sparse activation, where only a subset of the model's total parameters is used for any given input, leading to massive model capacity with efficient inference. MoE is a prime example, but the principle also underpins methods like early exiting and dynamic routing.

A parameter-efficient fine-tuning (PEFT) strategy that updates only a small, strategically selected subset of a model's parameters. Unlike MoE's sparse activation, this focuses on sparse updates during training. Key methods include:

• Diff Pruning: Learns a sparse binary mask over the pre-trained weights.
• Selective Fine-Tuning: Identifies and updates only the most task-relevant parameters.
• BitFit: Updates only the bias terms in a transformer model.

A secondary neural network that generates the weights or parameters for a primary (main) model. In modular adaptation, a hypernetwork can dynamically produce the parameters for task-specific adapter modules or expert layers based on a conditioning input (e.g., a task ID). This decouples the number of trainable parameters from the size of the main model, enabling efficient multi-task adaptation.

A broad PEFT paradigm that extends a frozen base model with small, self-contained, and composable neural modules. These modules, such as adapters or experts, are tuned for specific tasks or skills. MoE is a form of modular adaptation where the modules (experts) are selected per token. Other techniques include AdapterFusion for combining adapter knowledge and AdapterSoup for ensembling adapters at inference.

The specialized sub-networks within a Mixture-of-Experts model. Typically, these are feed-forward networks (FFNs) that replace or augment the standard FFN in a transformer block. A gating network routes each input token to the top-k most relevant experts. The capacity of an MoE model scales by adding more experts without increasing computational cost proportionally, as only a few are active per token.

A technique for creating a Mixture-of-Experts model by converting a dense, pre-trained model's feed-forward layers into expert layers. Instead of training experts from scratch, it reuses the existing weights of the dense model's FFNs, initializing multiple experts from them. This provides a strong starting point for the MoE model, improving training stability and final performance compared to random initialization.