Capacity Factor

The capacity factor is a multiplier applied to the base expert capacity to create a soft buffer, preventing token drops during routing. The base capacity for each expert is calculated as: (batch_size * sequence_length * k) / num_experts where k is the number of experts selected per token (e.g., top-2). The final capacity per expert is this value multiplied by the capacity factor (typically 1.0 to 2.0). This buffer accounts for the natural imbalance in token distribution, ensuring all routed tokens can be processed even if the load is not perfectly uniform.

The core purpose of the capacity factor is to act as a safety margin against dropped tokens. Without a buffer (capacity_factor = 1.0), if more tokens are routed to an expert than its base capacity, the excess tokens are 'dropped'—they bypass the expert computation entirely, often being passed to the next layer unchanged or via a fallback. This degrades model quality. A factor >1.0 (e.g., 1.25) allocates extra 'slots' per expert, drastically reducing drop rates and preserving model fidelity, albeit at a small increase in compute cost.

The capacity factor sits at the center of a key engineering trade-off:

• Low Factor (~1.0): Maximizes compute efficiency and throughput. Experts operate at or near 100% theoretical utilization, but risks high drop rates and potential model degradation.
• High Factor (~1.5-2.0): Maximizes model fidelity by virtually eliminating token drops. However, it reduces efficiency because experts are underutilized; they allocate memory and perform computations on padded, empty slots. The optimal value is found empirically, balancing acceptable drop rate against compute budget.

The capacity factor works in concert with load balancing techniques during training. An auxiliary load balancing loss encourages the router to distribute tokens evenly. A well-balanced router reduces the variance in expert assignment, meaning a lower capacity factor can be used safely. Conversely, poor load balancing necessitates a higher factor to accommodate the 'hot' experts receiving disproportionate traffic. Thus, the required factor is a direct reflection of the router's training quality and the inherent spikiness of the input distribution.

In expert parallelism, where experts are sharded across multiple GPUs, the capacity factor directly influences communication volume and memory allocation. A higher factor increases the size of the all-to-all communication buffers used to send tokens to their assigned expert's device. It also dictates the static allocation of memory for expert inputs/outputs on each device. Systems like vLLM with MoE support must manage this padded capacity efficiently within their paging and batching systems to avoid wasted memory and communication overhead.

Tuning the capacity factor is a standard step in MoE inference optimization. The process involves:

1. Profiling token distribution across experts for a target workload.
2. Measuring the token drop rate at different factor values.
3. Benchmarking the end-to-end latency and throughput impact. A typical workflow starts with a factor of 1.25 or 1.5, then iteratively reduces it while monitoring drop rates and downstream task accuracy (e.g., perplexity). The goal is to find the minimum factor where drop rates are negligible (<0.1%), ensuring no quality regression.

The maximum number of tokens that can be assigned to a single expert in a Mixture of Experts forward pass. It is a hard limit set during model implementation.

• Directly related to Capacity Factor: The capacity factor is a multiplier applied to the base expert capacity calculation to provide a buffer.
• Purpose: Prevents a single expert's computation from becoming a bottleneck by batching token computations, but setting it too low risks dropped tokens.
• Calculation: Base capacity is often (batch_size * sequence_length * k) / num_experts. The capacity factor (e.g., 1.25) is multiplied by this to set the final, padded capacity.

The routing strategy where the gating network selects the top k experts with the highest scores for each input token.

• Defines Sparsity: The value of k (e.g., 1 in Switch Transformers, 2 in Mixtral) determines how many experts are activated per token, directly controlling conditional computation.
• Impacts Capacity Planning: The k value is a key variable in the base expert capacity formula (batch_size * seq_len * k) / num_experts. A higher k increases the minimum required capacity per expert.
• Contrast with Routing: This is the specific algorithm that performs the expert routing decision.

A set of techniques to ensure the router distributes tokens evenly across all experts, preventing a small subset from being overloaded while others are underutilized.

• Why it's needed: Without it, the router can collapse, always selecting the same few experts. This makes expert capacity a bottleneck for those few experts and wastes the parameters of idle experts.
• Implementation: Often enforced via an auxiliary loss (load loss) added during training, which penalizes the variance in token assignment across experts.
• Relation to Capacity Factor: Good load balancing makes capacity planning more predictable and efficient, as token distribution is uniform, reducing the risk of localized capacity overflows.

Tokens that cannot be processed by their initially assigned expert because that expert has reached its expert capacity limit.

• Primary Cause: Occurs when the number of tokens routed to an expert exceeds its fixed capacity. The capacity factor is a hyperparameter tuned specifically to minimize this occurrence.
• Handling Strategies: Common fallbacks include passing the token unchanged to the next layer (a skip connection) or re-routing it to a secondary, less-loaded expert.
• System Impact: Dropped tokens represent a failure in the routing-computation pipeline and can degrade model performance, making their minimization a key systems optimization goal.

A collective communication pattern essential for expert parallelism, where devices exchange tokens and computed results after routing decisions.

• Process Flow: 1) All devices compute router scores for their local tokens. 2) Tokens are scattered (sent) to the specific devices hosting their assigned experts. 3) Experts compute. 4) Results are gathered back to the original devices.
• Performance Bottleneck: This communication can dominate latency in distributed MoE inference. Efficient capacity planning (via the capacity factor) ensures experts on each device receive balanced workloads, making the all-to-all exchange more efficient.

The fundamental property of MoE and other conditional computation models where only a small, dynamically chosen subset of the model's total parameters is used for a given input.

• Core Efficiency Proposition: Enables models with vastly more parameters (e.g., trillions) while keeping the computational cost per token relatively low, proportional to the active subset.
• Enabled by Routing: The gating network and top-k gating are the mechanisms that enable sparse activation.
• System Challenge: Sparse activation creates irregular, data-dependent computation graphs. The capacity factor and expert capacity are levers to make this irregularity manageable for efficient hardware execution (e.g., via fused MoE kernels).