Mixture of Experts (MoE)

Unlike dense models where every parameter participates in every forward pass, MoE models activate only a small fraction of total parameters per token. A gating network selects the top-k experts (typically 1 or 2) for each input, leaving the remaining experts idle. This conditional computation enables training models with trillions of parameters while keeping the FLOPs per token roughly constant.

• Token-level routing: Each token in a sequence is independently routed
• Top-k gating: Only the k highest-scoring experts process the token
• Load balancing: Auxiliary loss functions prevent expert collapse

The gating network (or router) is a lightweight, trainable component that determines which experts receive each token. It computes a probability distribution over all experts using a softmax function, then selects the top-k. The router is trained jointly with the experts via backpropagation.

• Softmax gating: Produces a normalized probability vector over experts
• Noisy top-k: Adds Gaussian noise during training to encourage exploration
• Expert capacity: Limits tokens per expert to prevent buffer overflow in distributed training

Each expert is a fully independent feed-forward network (FFN) with its own weights. Through training, experts naturally specialize in different aspects of the data distribution—for clinical models, one expert might handle cardiology terminology while another processes oncology notes. This emergent specialization occurs without explicit topic assignment.

• Domain-specific fine-tuning: Experts can be further adapted to subdomains
• Knowledge partitioning: Implicit division of the input manifold
• Expert dropout: Regularization technique to prevent co-adaptation

Without intervention, the gating network may collapse into routing all tokens to a single expert, defeating the purpose of the architecture. An auxiliary loss function penalizes imbalanced routing, encouraging uniform expert utilization. This is critical for maintaining throughput in distributed training where each expert resides on a separate device.

• Auxiliary loss: Penalizes deviation from uniform expert usage
• Z-loss: Stabilizes training by penalizing large router logits
• Expert capacity factor: Hard limit on tokens per expert per batch

MoE models are inherently suited for model parallelism. Each expert can be placed on a dedicated accelerator (GPU/TPU), with all-to-all communication occurring only for token dispatch and result collection. This enables scaling to models far larger than what fits on a single device.

• Expert parallelism: Each expert resides on a separate device
• All-to-all collectives: Tokens are redistributed across devices per layer
• Pipeline parallelism: Layers are also distributed across devices for further scaling

For healthcare language models, MoE architectures offer distinct benefits. The specialized expert structure aligns naturally with the heterogeneous nature of clinical text—radiology reports, operative notes, and discharge summaries each have distinct linguistic patterns. A single MoE model can maintain high accuracy across all document types without cross-task interference.

• Multi-specialty coverage: Different experts handle different medical domains
• Catastrophic forgetting resistance: New medical knowledge can be added via expert expansion
• Efficient inference: Only relevant experts activate for a given clinical document type

The discrete routing mechanism that selects which experts to activate for a given input token. The router computes a probability distribution over all experts and typically selects the top-k (e.g., top-2) experts via a softmax gate. A load-balancing loss is crucial to prevent the network from collapsing into a state where only a few experts are utilized, ensuring all parameters are trained effectively.

A critical hyperparameter that defines the maximum number of tokens a single expert can process per batch. It is calculated as (tokens_per_batch / num_experts) * capacity_factor. If an expert's assigned tokens exceed this capacity, the overflow is dropped (residual connection bypass). This enforces strict computational budgeting, preventing latency spikes from overloaded experts.

The standard formulation where only a sparse subset of experts is activated per token, contrasting with dense MoE. Key implementations include:

• Switch Transformer: Simplifies routing to a single expert (top-1) to reduce communication costs.
• GShard: A top-2 gating architecture designed for massive scale, demonstrating the viability of MoE for trillion-parameter models.
• Expert Choice: Inverts the routing paradigm, letting experts select the top-k tokens.

An auxiliary loss function added to the primary task loss to prevent representation collapse. Without it, the gating network can converge to a degenerate state where it always routes tokens to a single, high-bias expert. The loss penalizes uneven expert assignment, typically by minimizing the variance of the expert assignment distribution or maximizing the entropy of the routing probabilities.

The primary distributed training strategy for MoE models. Since experts are independent feed-forward networks, they can be placed on disparate accelerator devices. All-to-all communication collects tokens destined for the same expert onto a single device, computes the expert function, and then redistributes the outputs back to the original sequence ordering. This is distinct from tensor or data parallelism.

A fully-differentiable alternative to discrete token routing that avoids the pitfalls of load balancing loss and token dropping. Instead of assigning tokens to specific experts, Soft MoE computes weighted combinations of all input tokens as expert inputs and similarly combines expert outputs. This maintains sparsity in the number of expert computations but ensures smooth gradients and balanced utilization.