Multi-gate Mixture-of-Experts (MMoE)

The foundational layer consists of multiple independent expert networks, typically implemented as multi-layer perceptrons (MLPs) with ReLU activations. These experts are shared across all tasks but are not forced to be used uniformly. Each expert can specialize in learning distinct patterns from the input data, such as user-item interaction dynamics, temporal behavior sequences, or demographic correlations, without explicit programming.

For each task k, a dedicated gating network is trained. This is a lightweight function, often a single linear layer followed by a softmax activation, that takes the same input features and outputs a probability distribution over the experts. The final representation for task k is a weighted sum of the expert outputs, where the weights are the gating probabilities. This allows the model to learn that Task A (e.g., CTR) relies heavily on Expert 1 and 3, while Task B (e.g., CVR) uses Expert 2 and 4.

A core failure mode of hard-parameter-sharing MTL is negative transfer, where optimizing for one task degrades performance on another due to conflicting gradient updates. MMoE resolves this by allowing tasks to ignore experts that are detrimental to them. If an expert learns a feature representation that helps predict clicks but hurts conversion prediction, the CVR gate can simply assign it a weight near zero, effectively decoupling the tasks while still benefiting from shared computational resources.

After the gating network computes the task-specific weighted expert combination, the resulting vector is passed to a task-specific tower network. This tower is a deep neural network (often 2-3 layers) that transforms the combined representation into the final prediction logit. The tower allows for further task-specific non-linear transformations before the output layer, which applies a sigmoid function for binary classification tasks like click probability.

MMoE is trained end-to-end using a joint loss function that is a weighted sum of the individual task losses. For CTR and CVR prediction, this is typically the sum of their binary cross-entropy losses: L_total = w_ctr * L_ctr + w_cvr * L_cvr. The loss weights are hyperparameters that control task prioritization. Backpropagation flows through the towers, gates, and experts simultaneously, allowing the gates to learn optimal routing strategies based on the final task objectives.

In classic hard parameter sharing, all tasks share the exact same hidden layers, forcing a single unified representation. This is highly susceptible to negative transfer. MMoE's innovation is the soft parameter sharing via gating. It maintains a shared pool of experts for computational efficiency but provides the flexibility of task-specific combination. This makes MMoE significantly more robust than hard sharing when task correlations are low or complex, a common scenario in real-world recommender systems.