Federated Mixture of Experts (MoE)

Unlike dense models where every parameter fires for every input, Federated MoE activates only a top-k subset of experts per query. This conditional computation means a hospital specializing in cardiology trains an expert that only activates for cardiac-related cases, dramatically reducing inference compute while maintaining broad clinical coverage.

• Token-level routing: Each token in a clinical note can be dispatched to a different expert
• Load balancing: Auxiliary loss functions prevent all queries from collapsing to a single popular expert
• Capacity factor: Limits how many tokens an expert processes per batch, enforcing true specialization

The gating network is the central coordination layer that learns to route incoming clinical queries to the most appropriate locally-trained expert. Trained on aggregate routing statistics rather than raw data, it learns that Institution A's expert handles pediatric oncology while Institution B's expert specializes in geriatric cardiology.

• Softmax routing: Produces a probability distribution over all available experts
• Noisy top-k gating: Adds tunable Gaussian noise during training to encourage exploration of expert assignments
• Auxiliary loss: Penalizes imbalanced expert utilization to ensure all institutional experts receive sufficient training signal

Each participating hospital trains one or more expert modules exclusively on its local patient population. A cancer center's expert becomes finely tuned to oncological pathology reports, while a rural clinic's expert specializes in primary care triage. The base model layers and gating network are shared, but expert weights never leave their originating institution.

• Domain-specific emergent experts: Specialization emerges naturally from local data distributions without manual assignment
• Expert capacity isolation: A surge in COVID-19 cases at one hospital only updates its local expert, preventing catastrophic forgetting in others
• Heterogeneous expert architectures: Different institutions can deploy experts with varying layer counts or attention heads based on local compute availability

Only the routing probabilities and aggregated expert utilization statistics are shared with the central gating network. Expert weights remain behind institutional firewalls. When federated fine-tuning is required, techniques like Federated LoRA or Federated Knowledge Distillation update experts without exposing the underlying clinical data or full model weights.

• Differential privacy guarantees: Gaussian noise can be added to routing statistics before transmission
• Secure aggregation: Multi-party computation protocols ensure the central server only sees the aggregated routing update, not individual institutional contributions
• Expert dropout: Randomly omitting experts during training forces the gating network to build robust, redundant routing pathways

A critical challenge in Federated MoE is preventing expert collapse, where the gating network routes all queries to a single high-performing institution. Auxiliary load-balancing losses and expert capacity limits ensure equitable utilization across the federated network, maintaining the value proposition for all participating hospitals.

• Importance-weighted balancing: Weights expert contributions by both utilization frequency and the magnitude of their routing probabilities
• Expert buffer capacity: Each expert is allocated a fixed token capacity per batch, with overflow tokens passed to the next available expert or dropped
• Dynamic expert scaling: New institutions can join the federation and contribute a new expert module without retraining the entire gating network

Federated MoE accommodates asymmetric institutional resources. A large academic medical center can train a deep expert with 12 transformer layers on its GPU cluster, while a community clinic trains a compact 4-layer expert on a single edge server. The gating network learns to account for these capability differences when routing queries.

• Expert capacity tagging: Each expert registers its computational profile and domain specialty with the central registry
• Latency-aware routing: Time-sensitive emergency department queries can be routed to low-latency edge experts
• Federated expert pruning: Less-utilized expert parameters can be pruned locally to reduce memory footprint without central coordination

The learned routing mechanism that decides which expert sub-models to activate for a given input. In a federated MoE, the gating network is typically the shared, centrally-aggregated component that learns to route clinical queries to the most relevant institutionally-trained expert.

• Uses a softmax layer to produce a probability distribution over experts
• Often employs top-k sparsity, activating only the 2-3 most relevant experts per token to save compute
• The gating function must be trained on aggregated routing statistics from all nodes without seeing raw data
• Load balancing losses prevent the network from collapsing to always selecting the same expert

The phenomenon where individual sub-models in the MoE naturally become domain-specific processors during training. In a federated healthcare context, one institution's expert may specialize in radiology reports while another becomes proficient at pathology terminology.

• Specialization emerges from the heterogeneous data distributions across hospitals
• Each expert is a full feed-forward network within the larger model, not just a classifier
• Experts can be conditionally activated based on input type, reducing inference cost
• Expert collapse is a failure mode where all inputs route to a single expert, negating the MoE benefit

The foundational aggregation algorithm that combines locally-trained model updates into a global model. In Federated MoE, FedAvg is applied separately to the shared gating network and to each expert module, with experts potentially aggregated only across institutions that contributed to their training.

• Each hospital trains on local data and sends model weight updates (not data) to the central server
• The server computes a weighted average of updates, typically weighted by local dataset size
• Communication rounds repeat until convergence
• FedProx is a variant that adds a proximal term to handle statistical heterogeneity across hospitals

A cryptographic guarantee that the aggregated model does not reveal information about any single patient's record. In Federated MoE, DP is applied during local training by clipping gradients and adding calibrated Gaussian noise before transmission.

• Controlled by the privacy budget (ε, delta) — lower epsilon means stronger privacy
• The DP-SGD algorithm modifies the standard training loop to bound individual contributions
• A critical tradeoff: tighter privacy guarantees typically reduce model utility
• Essential for HIPAA and GDPR compliance in cross-institutional healthcare AI

The statistical reality that clinical data across hospitals is not independent and identically distributed. This is both a challenge and the primary motivation for Federated MoE, as different data distributions naturally drive expert specialization.

• Label distribution skew: Hospital A has more cardiac cases, Hospital B more oncology
• Feature distribution skew: Different imaging equipment produces varying pixel intensity distributions
• Concept drift: The same diagnosis may be documented differently across institutions
• Federated MoE explicitly models this heterogeneity rather than fighting it, assigning experts to distinct data regimes

A cryptographic protocol that allows the gating network to route queries to the correct expert without any party, including the central server, learning which institution's expert was selected. This protects institutional competitive advantage in addition to patient privacy.

• Uses secret sharing to split routing decisions across multiple non-colluding servers
• Oblivious transfer protocols enable querying an expert without revealing which one
• Adds significant communication overhead compared to non-secure routing
• An active research area for making Federated MoE deployment-ready in competitive healthcare networks