Mixture of Experts (MoE)

The core computational sub-networks in a MoE layer. Each expert is typically a feed-forward neural network (FFN) with its own unique set of parameters. In transformer-based MoEs, experts replace the standard FFN layer. The model's total parameter count is massive, but for any given input, only a small, sparse subset of experts is activated, keeping the computational cost (FLOPs) manageable. Experts are designed to specialize in different types of features or data patterns during training.

The intelligent dispatcher of the MoE system. The gating network is a small, trainable neural network (often a simple linear layer) that receives an input token's hidden state and outputs a set of scores or logits, one for each expert. These scores determine which experts are best suited to process the token. The router's primary function is expert routing, enabling dynamic, input-dependent computation. Its design (e.g., top-k, noisy top-k) is critical for load balancing and model performance.

The algorithm that translates router scores into discrete token assignments. The most common strategy is Top-k Gating, where only the k experts with the highest scores are selected per token. For example, Mixtral 8x7B uses top-2 gating. Variants include:

• Noise Top-k Gating: Adds tunable Gaussian noise to logits before selection to encourage exploration and better load balancing during training.
• Switch Routing (Top-1): A simplified, highly scalable approach where each token is routed to exactly one expert.
• Hard Routing: The general paradigm of discrete assignment, as opposed to soft blending of all expert outputs.

A critical auxiliary mechanism to prevent router collapse, where the gating network favors a small subset of experts, leaving others underutilized. This is enforced via an Auxiliary Loss (Load Loss) added to the main training objective. Common formulations, like the one in Switch Transformers, minimize the squared coefficient of variation of the routing probability distribution across experts. Proper load balancing ensures uniform expert utilization, maximizing the effective use of the model's vast parameter space and maintaining training stability.

A key hyperparameter that governs system throughput and token dropping. Expert capacity is the maximum number of tokens that can be assigned to a single expert in a forward pass. It is calculated as: Capacity = (batch_size * sequence_length * k) / num_experts * capacity_factor The capacity factor (typically 1.0 to 2.0) provides a buffer above this theoretical minimum. If an expert receives more tokens than its capacity, the excess tokens are dropped and may be passed through unchanged or handled by a fallback. A higher factor increases memory usage but reduces drop rate.

The system-level orchestration required for distributed MoE execution. Expert Parallelism is a model parallelism strategy where different experts are placed on different devices (e.g., GPUs). This necessitates efficient All-to-All Communication: after local routing decisions, tokens are scattered across the device mesh to their assigned expert's device, and the processed outputs are gathered back. This communication overhead is a major bottleneck. Optimized Fused MoE Kernels combine routing and computation into single GPU operations to minimize latency.

This is not a model but a critical distributed training strategy for MoE. It addresses the memory constraints of housing massive experts on a single device.

• Mechanism: Experts are sharded across different GPUs. After the router assigns tokens to experts, an all-to-all communication collective scatters tokens to the devices hosting their assigned experts.
• Bottleneck: The all-to-all operation can become a network bandwidth bottleneck, making high-speed interconnects (e.g., NVLink, InfiniBand) essential for training large MoE models efficiently.
• Frameworks: Implemented in deep learning frameworks like Megatron-LM and DeepSpeed.

The defining property of conditional computation models like Mixture of Experts, where only a small, dynamically selected subset of the model's total parameters is used for a given input. This contrasts with dense models, where every parameter participates in every forward pass.

• Key Benefit: Enables massive model scale (e.g., trillions of parameters) without a proportional increase in computational cost per token.
• Core Mechanism: Achieved via a gating network that routes each token to specific experts.

A small, trainable neural network component that determines expert routing. It receives an input token's hidden state and outputs a set of scores or probabilities used to select which expert(s) should process it.

• Function: Implements the routing policy (e.g., Top-k, Switch).
• Training Challenge: Must be co-trained with the experts, often requiring auxiliary loss functions to prevent collapse and ensure load balancing.
• Latency Impact: Adds a small but non-negligible router latency to each layer's computation.

The two predominant strategies for hard routing tokens to experts.

• Top-k Gating: Selects the k experts with the highest router scores for each token. The outputs of these k experts are combined via a weighted sum based on their scores. Common in models like Mixtral (k=2).
• Switch Routing (Top-1): A simplification where each token is routed to the single highest-scoring expert (k=1). Pioneered by the Switch Transformer, it simplifies communication and implementation.
• Noise Top-k: A variant that adds tunable Gaussian noise to router logits before selection, encouraging exploration and better load balancing during training.

Critical techniques to prevent router collapse, where a few experts are overused while others are starved.

• The Problem: Without regularization, the router can develop a strong preference for a subset of experts, undermining the benefits of sparsity and causing computational hotspots.
• Auxiliary Loss (Load Loss): An additional term added to the primary training objective. A common formulation, as used in Switch Transformers, minimizes the squared coefficient of variation of the routing probability distribution across experts.
• Outcome: Encourages uniform token distribution, ensuring all experts are trained and utilized effectively.

The model parallelism strategy and communication pattern essential for distributing large MoE models across multiple devices (e.g., GPUs).

• Expert Parallelism: Experts are sharded across devices. Unlike tensor or pipeline parallelism, this distribution is conditional—the device a token visits depends on the router's decision.
• All-to-All Communication: After local routing, tokens must be sent (scattered) from all devices to the specific devices hosting their assigned experts. After expert computation, the resulting embeddings are gathered back. This communication is often the system bottleneck.
• Optimization: Requires careful overlap of computation and communication to hide latency.

Production systems and low-level optimizations designed to serve MoE models efficiently.

• vLLM with MoE Support: An inference serving engine that extends its PagedAttention and continuous batching optimizations to handle MoE's sparse computation and variable memory patterns, improving throughput.
• Fused MoE Kernels: Custom GPU kernels that combine routing logic, token permutation, and the sparse matrix multiplications for multiple experts into a single operation. This minimizes kernel launch overhead and expensive memory movements between routing and computation phases.
• Continuous Batching for MoE: Adapts dynamic batching techniques to handle the irregular computational graphs created by per-token routing, improving GPU utilization.