Hard Routing

Hard routing makes a deterministic selection of one or more experts per token, typically via an argmax or top-k operation on the router's logits. This creates a sparse computational graph where only the weights of the selected experts are activated. For example, in a top-2 routing policy, each token's hidden state is processed by exactly two expert networks, and their outputs are combined (e.g., summed). This is fundamentally different from soft routing, where outputs might be blended using continuous attention-like scores.

The most common hard routing mechanism is Top-k Gating. A lightweight gating network (router) computes a score for each expert. For each input token, the router selects the k experts with the highest scores. In practice, k is small (1 or 2) to maintain sparsity. The process is:

• Compute router logits for the token.
• Select top k experts.
• Route the token's hidden state only to those experts.
• Compute and combine the expert outputs. This ensures a fixed, predictable FLOPs cost per token, proportional to k, not the total number of experts.

Hard routing enables conditional computation, activating only a fraction of the model's total parameters. This allows for gigantic models (e.g., trillion-parameter) with feasible inference costs. However, it introduces significant system complexity:

• Load Balancing: The router can develop favorites, causing some experts to be overloaded while others are idle. This is mitigated by auxiliary loss functions during training.
• All-to-All Communication: In expert-parallel distributed training and inference, tokens must be scattered to the devices hosting their assigned experts and gathered afterward, creating a communication bottleneck.
• Expert Capacity: Each expert has a fixed capacity (number of tokens it can process per batch). Tokens exceeding an expert's capacity are dropped, potentially harming model quality.

Through training, hard routing encourages token-expert affinity, where experts naturally specialize in processing specific types of tokens. For instance, in a language model, one expert might specialize in mathematical symbols, another in proper nouns, and another in syntactic function words. This specialization emerges because the router learns to assign semantically or syntactically similar tokens to the same subset of experts, making the overall model more compositional and efficient. The discrete nature of the assignment reinforces this specialization more sharply than soft routing.

Hard routing is defined in opposition to soft routing. Key differences:

• Hard Routing: Discrete, sparse selection (e.g., top-k). Outputs are combined via a weighted sum based on the router scores for the selected experts only.
• Soft Routing: Continuous, dense blending. All experts receive the token, and their outputs are combined using a full set of attention-like weights, resulting in a denser, more computationally expensive operation. Hard routing is preferred in large-scale MoE (e.g., Switch Transformer, Mixtral) for its computational sparsity. Soft routing is more common in older, smaller mixture-of-experts models and is analogous to multi-head attention.

Several landmark models exemplify hard routing:

• Switch Transformer: Uses top-1 routing (Switch Routing), where each token is sent to exactly one expert. This maximizes sparsity and simplifies system design.
• Mixtral 8x7B: Employs top-2 routing. Each layer has 8 expert FFNs, and each token is processed by 2, giving it the effective capacity of a ~47B parameter dense model for only ~13B active parameters.
• GLaM (Google): A generalist language model using top-2 routing with a massive 1.2 trillion total parameters, demonstrating the scalability of the paradigm. These models rely on fused MoE kernels and optimized expert parallelism to make the routing and sparse computation efficient on GPU clusters.

Mistral AI's Mixtral 8x7B is a state-of-the-art open-source model that uses hard top-2 routing. For each token, the router selects the two highest-scoring experts out of eight per layer. Despite having 47B total parameters, it activates only about 13B parameters per token, matching the inference cost of a much smaller dense model. This architecture demonstrates how hard routing enables high-quality, cost-effective inference for open-weight LLMs.

• Model Family: Mixtral 8x7B, Mixtral 8x22B
• Routing Policy: Top-2
• Key Benefit: Dense 13B model latency with 47B parameter knowledge.

A core challenge in training hard-routed MoEs is load imbalance, where the router may collapse to always selecting the same few experts. To prevent this, an auxiliary load balancing loss is added during training. For top-2 routing, this loss encourages the mean routing probability across experts to be uniform and minimizes the squared coefficient of variation of the number of tokens assigned to each expert. This technique is critical for ensuring all 47B parameters in a model like Mixtral are effectively utilized.

The choice between hard and soft routing has direct engineering implications:

Hard Routing (e.g., Top-k)

• Pros: Sparse, deterministic computation; efficient expert parallelism; lower memory bandwidth (no blending coefficients).
• Cons: Discontinuous gradients; requires load balancing losses; risk of dropped tokens if expert capacity is exceeded.

Soft Routing (e.g., Mixture)

• Pros: Continuous, differentiable operation; natural load balancing.
• Cons: Dense computation (all experts are weighted); higher memory and compute cost; complex to scale.

In production LLM serving, hard routing dominates due to its superior computational efficiency and predictable latency.

A neural network architecture where the model is composed of multiple sub-networks (experts), and a routing mechanism dynamically selects a sparse subset of these experts to process each input. This enables massive parameter counts (e.g., trillions) with a conditional, much lower computational cost per token, as only a few experts are active for any given input.

A small, trainable neural network component within a MoE model that receives an input token or hidden state and outputs a set of scores or logits. This network is responsible for the routing decision. In hard routing paradigms, it uses functions like argmax or top-k to make discrete, non-differentiable assignments of tokens to specific experts.

The most common hard routing strategy. For each input token, the gating network computes scores for all experts, selects the k experts with the highest scores, and routes the token's representation exclusively to those experts. The outputs of the selected experts are then combined, typically via a weighted sum based on the gating scores. This enforces a fixed, predictable computational budget per token.

A critical challenge in MoE training. Without intervention, the router can collapse, always selecting the same few popular experts. Load balancing techniques prevent this, ensuring uniform computational load. Key methods include:

• Auxiliary Loss (Load Loss): An added training loss term that penalizes imbalance.
• Noise Top-k Gating: Adds tunable noise to router logits to encourage exploration.
• Expert Capacity: A hard limit on tokens per expert per batch.

A model parallelism strategy designed for MoE models at scale. Different expert networks are placed on different devices (e.g., GPUs). The routing decision triggers an all-to-all communication pattern:

1. Tokens are scattered from all devices to the devices hosting their assigned experts.
2. Experts process their assigned tokens in parallel.
3. Results are gathered back to the original devices. This is essential for distributing the massive parameter count of MoE models across a hardware cluster.

The defining computational property enabled by hard routing. In a sparsely activated MoE model, only a small, dynamically chosen subset of the model's total parameters (the selected experts) is involved in the forward pass for a given input. This contrasts with dense models (like standard transformers), where every parameter is used for every input. Sparse activation is the key to the efficiency vs. parameter count trade-off.