Dropped Tokens

The primary cause of dropped tokens is exceeding an expert's capacity, a fixed limit set for computational batching. The capacity factor (e.g., 1.25) provides a buffer:

• Capacity = (batch_size * seq_len * k / num_experts) * capacity_factor.
• If more tokens are routed to an expert than its capacity, the excess tokens are dropped. This is a deliberate engineering trade-off to maintain uniform tensor sizes for efficient GPU computation, accepting a small performance degradation to avoid dynamic, irregular memory allocation.

A common handling strategy where dropped tokens are not discarded but rerouted. Typically, they are passed to a designated fallback expert (often the most capacious or a shared expert).

• This ensures all tokens receive some processing, preventing complete information loss.
• The fallback expert acts as a computational safety net, but its generalized processing may be less optimal than the router's original, specialized choice.

The most straightforward handling method: a dropped token bypasses the expert layer entirely via its residual connection.

• The token's hidden state passes to the next transformer layer unchanged.
• This treats the expert's feed-forward network as a conditional refinement; if unavailable, the base representation persists.
• While simple, this can degrade model quality if drops are frequent, as specialized transformations are missed.

A preventive strategy applied during training. An auxiliary loss (e.g., load balancing loss) penalizes the router for uneven token distribution.

• This encourages the gating network to spread tokens more evenly across experts.
• By promoting balanced utilization, it minimizes the probability that any single expert exceeds its capacity during inference, thereby reducing dropped tokens at their source.

Advanced routing mechanisms that consider real-time expert load.

• Algorithms like Capacity-constrained Top-k or Dynamic MoE adjust routing decisions on-the-fly to avoid sending tokens to already-full experts.
• This moves beyond static capacity limits, enabling more adaptive and efficient token assignment, though it adds complexity to the routing logic and communication pattern.

Dropped tokens represent a key system-level trade-off in MoE inference between computational efficiency and model quality.

• High capacity factor: Fewer drops, higher quality, but increased memory overhead and potential underutilization (experts process padded, empty slots).
• Low capacity factor: Higher memory efficiency and denser computation, but increased drop rate and potential quality degradation.
• Tuning this is critical for production latency and cost targets.

The maximum number of tokens that can be assigned to a single expert in a Mixture of Experts forward pass. It's a critical implementation parameter set as a multiple of the average expected load (the capacity factor).

• Balancing Act: A low capacity minimizes wasted computation on padding but increases dropped token risk. A high capacity ensures all tokens are processed but adds computational overhead from padding.
• Implementation: Often implemented as a fixed-size buffer per expert. Once an expert's buffer is full, subsequent tokens routed to it become dropped tokens.

A set of techniques designed to prevent the router from consistently favoring a small subset of experts, ensuring uniform computational load. Poor load balancing is a primary cause of dropped tokens, as popular experts quickly hit their capacity limit.

• Auxiliary Loss (Load Loss): An additional term added during training that penalizes the variance in token assignment across experts, encouraging the router to distribute tokens more evenly.
• Noise Top-k Gating: A routing variant that adds tunable Gaussian noise to expert logits before selection, acting as an exploration mechanism during training to discover a better balanced routing policy.

A hyperparameter that defines the expert capacity as a multiple of the average expected load per expert. It provides a buffer to handle token distribution variance and prevent dropped tokens.

• Calculation: Expert Capacity = Capacity Factor * (batch_size * sequence_length * k) / num_experts
• Typical Values: Commonly set between 1.0 and 2.0. A factor of 1.0 means capacity equals the average load, leaving no buffer for imbalance. A factor of 1.25 provides a 25% buffer.
• Trade-off: Higher factors increase GPU memory usage and computational waste on padded slots but reduce the probability of token dropping.

The fundamental property of conditional computation models like Mixture of Experts, where only a small, dynamically chosen subset of the model's total parameters is activated for a given input. Dropped tokens represent a breakdown in this sparse execution paradigm.

• Contrast with Dense Models: In a dense model (e.g., a standard Transformer), all parameters are used for every token, leading to predictable but higher fixed cost.
• Efficiency Goal: Sparse activation aims to achieve the quality of a larger model at the computational cost of a smaller one. Dropped tokens undermine this by causing some inputs to bypass the intended expert computation.

A collective communication pattern critical to Expert Parallelism, where model experts are distributed across multiple devices (GPUs). This pattern directly influences the handling of capacity and dropped tokens in distributed settings.

• Process: After the routing decision on each device, tokens destined for experts on other devices are sent (scatter). After processing, the expert outputs are gathered back (gather).
• System Challenge: The volume of this communication is determined by the routing imbalance. Poor load balancing can cause network hotspots, mimicking the capacity limits that cause dropped tokens within a single device.

A designated expert or computational path used to process dropped tokens when their primarily assigned expert is at capacity. This is a common mitigation strategy to avoid passing the token through unchanged.

• Implementation: Can be a shared, lightweight expert, a copy of a less-loaded expert, or a simple feed-forward network.
• Trade-off: While it prevents information loss, it adds system complexity and may degrade quality if the fallback expert is not as capable as the primary choice.
• Alternative: Some systems simply pass the dropped token's input representation directly to the next layer (a residual passthrough), which is simpler but assumes the layer's transformation is non-critical.