Multi-Task Learning

The foundational MTL architecture where a shared hidden layer stack is followed by task-specific output heads. All tasks jointly update the shared weights during backpropagation, forcing the network to learn features that are universally useful. This drastically reduces the risk of overfitting to any single task by acting as a strong regularizer, but can suffer from negative transfer if tasks are too dissimilar.

An alternative architecture where each task has its own model with its own parameters, but the distance between parameters is regularized. Techniques like L2 distance or trace norm are applied to encourage the separate models to have similar weights. This mitigates the optimization conflicts of hard sharing, allowing for more task-specific feature extraction while still enabling cross-task knowledge transfer.

A soft-sharing mechanism that learns a linear combination of activation maps from multiple single-task networks. Cross-stitch units are trainable parameters that determine the degree of sharing between tasks at each layer. This allows the model to dynamically learn a task-sharing policy, discovering which features should be shared and which should remain private without manual architecture engineering.

A training optimization technique that resolves conflicting gradients between tasks. When the gradients for two tasks point in opposing directions, Projecting Conflicting Gradients (PCGrad) projects each gradient onto the normal plane of the other. This prevents the destructive interference that causes one task's performance to degrade while another improves, stabilizing the joint optimization process.

A loss balancing strategy that treats task weights as learnable parameters based on homoscedastic task uncertainty. Instead of manually tuning fixed loss coefficients, the model learns to weigh tasks by the inherent noise in their predictions. Tasks with higher data noise are automatically down-weighted, preventing a noisy regression task from dominating a cleaner classification task during training.

A conditioning mechanism where each task is represented by a learned vector that modulates the shared network's behavior. The task embedding is concatenated with the input or used to scale and shift features via FiLM layers. This allows a single shared backbone to specialize its processing for different tissues or experimental conditions without requiring separate output heads for every single task.

The foundational MTL architecture where all tasks share the same hidden layers, with only task-specific output heads diverging at the top. Shared representations learn generalizable genomic features—such as transcription factor binding motifs—that benefit all tasks. This drastically reduces overfitting risk on small datasets by increasing the effective sample size for the shared layers. In gene expression prediction, a single convolutional trunk might process DNA sequence, while separate heads predict expression in liver, heart, and brain tissues simultaneously.

An MTL variant where each task maintains its own model parameters, but a regularization penalty encourages them to remain similar. Techniques include L2 distance constraints or cross-stitch networks that learn linear combinations of activations between task-specific networks. This architecture handles heterogeneous tasks better than hard sharing—useful when predicting both RNA-seq coverage and chromatin accessibility from the same sequence, as the optimal representations may partially diverge.

An adaptive loss weighting algorithm that dynamically adjusts task weights during training by monitoring gradient magnitudes. It prevents any single task from dominating optimization by penalizing tasks whose gradients are too large or too small relative to a target norm. In multi-tissue expression prediction, GradNorm automatically balances learning between highly expressed housekeeping genes and tissue-specific transcripts, preventing the former from overwhelming gradient updates.

A probabilistic approach that weights multiple loss functions by deriving optimal weights from each task's homoscedastic uncertainty—the task-dependent observation noise. The model learns to down-weight noisy or difficult tasks automatically. For genomic prediction, this means a model jointly predicting TPM-normalized expression and binary enhancer activity can learn appropriate relative scaling without manual hyperparameter tuning, treating each task's noise as a learnable parameter.

A soft parameter sharing architecture where separate networks for each task exchange information through learned linear combinations of activations at every layer. A cross-stitch unit learns a matrix that determines how much each task's representation should incorporate features from other tasks. Applied to genomics, this allows a model predicting CAGE-seq and RNA-seq signals to share convolutional features while maintaining task-specific pathways for their distinct statistical properties.