Multi-Task Learning

The foundational architectural pattern where all tasks share a common set of hidden layers, with only the final output layers branching into task-specific heads. This forces the network to learn a universal regulatory grammar that generalizes across assays.

• Mechanism: A single convolutional or transformer trunk processes the DNA sequence, then feeds into separate fully-connected heads for each TF or cell type.
• Benefit: Reduces overfitting on low-data tasks by regularizing through shared representations.
• Example: DeepSEA predicts 919 chromatin features from a single shared trunk, learning a compressed representation of the non-coding genome.

Each task maintains its own model parameters, but the distance between parameters is regularized using an L2 norm or trace norm penalty. This allows task-specific specialization while still encouraging representational similarity.

• Mechanism: Separate networks are trained jointly with a loss term penalizing weight divergence.
• Use Case: When tasks are loosely related—e.g., predicting binding for TFs from different structural families—soft sharing prevents negative transfer while still enabling cross-task learning.
• Contrast: Unlike hard sharing, this avoids forcing unrelated tasks into a single representational bottleneck.

Balancing the contribution of each task to the total loss function is critical. Unequal weighting can cause the model to overfit to high-variance tasks or ignore low-signal assays entirely.

• Uncertainty Weighting: Uses homoscedastic task uncertainty to dynamically adjust weights during training, treating each task's loss as a Gaussian likelihood.
• GradNorm: Adjusts weights based on the magnitude of gradients, ensuring all tasks train at similar rates.
• Static Weighting: Manually tuned hyperparameters based on validation performance, common in production pipelines where task priority is domain-driven.

A learnable linear combination of hidden activations from multiple single-task networks, controlled by a cross-stitch matrix. This unit sits between layers and learns the optimal degree of feature sharing for each task pair.

• Mechanism: Given two task networks, the cross-stitch unit computes a weighted sum of their activations at each layer, with weights learned via backpropagation.
• Advantage: Automatically discovers which tasks benefit from sharing and which should remain independent.
• Genomic Application: Useful when combining diverse assay types—e.g., TF binding, histone marks, and DNase-seq—where sharing relationships are not known a priori.

The phenomenon where multi-task learning degrades performance on individual tasks compared to single-task baselines. This occurs when tasks are too dissimilar or compete for representational capacity.

• Detection: Monitor per-task validation loss; if a task consistently underperforms its single-task counterpart, negative transfer is likely.
• Mitigation Strategies:
  • Task grouping: Cluster related tasks and train separate multi-task models per group.
  • Gradient surgery (PCGrad): Projects conflicting gradients onto orthogonal planes before updating.
  • Dynamic task pruning: Drop tasks from the shared trunk if their gradient cosine similarity falls below a threshold.

Multi-task architectures are particularly effective for low-N genomics, where individual transcription factors may have limited ChIP-seq training examples. Shared representations transfer knowledge from data-rich to data-poor assays.

• Transfer Learning Effect: A model trained on hundreds of TF binding profiles learns a generalizable 'sequence syntax'—motif grammar, spacing constraints, and chromatin context—that bootstraps prediction for novel factors.
• Zero-Shot Capability: Models like BPNet can predict binding for TFs never seen during training by leveraging shared motif representations learned from related factors.
• Practical Impact: Reduces the number of costly ChIP-seq experiments required to characterize a new transcription factor's binding landscape.

The dominant architectural paradigm in genomic multi-task learning where a shared hidden representation is learned by all tasks, with only task-specific output heads diverging. This forces the network to discover features that are universally predictive across assays.

• Reduces overfitting risk by a factor of N (number of tasks)
• Enforces a unified regulatory grammar in the latent space
• Example: A single convolutional trunk predicting ChIP-seq peaks for 100+ transcription factors

An alternative paradigm where each task maintains its own model parameters but a regularization penalty encourages similarity between them. Common techniques include L2 distance constraints on weights or trace norm regularization.

• Allows task-specific divergence when assays are heterogeneous
• Preserves capacity for assay-specific noise patterns
• Computationally more expensive than hard sharing

Methods for balancing the contribution of each assay's loss during training to prevent high-variance or noisy tasks from dominating gradient updates.

• Uncertainty weighting: Learns task-dependent homoscedastic uncertainty to automatically weight losses
• GradNorm: Dynamically adjusts weights to equalize gradient magnitudes
• Dynamic Weight Average: Balances learning pace across tasks by monitoring loss reduction rates

An architecture that learns a linear combination of hidden activations from multiple task-specific subnetworks at each layer. A cross-stitch unit is a trainable matrix that determines how much each task borrows from others.

• Enables task-specific feature hierarchies while enabling selective sharing
• Automatically discovers which layers benefit from shared representations
• Applied in genomics to jointly model DNase-seq, ATAC-seq, and ChIP-seq

The phenomenon where multi-task learning degrades performance relative to single-task baselines. Occurs when tasks are insufficiently related or when optimization conflicts arise.

• Detected when validation loss for a task increases with joint training
• Mitigated through task grouping or gradient surgery (PCGrad)
• Common when mixing fundamentally different assay modalities without sufficient shared biology

Projecting Conflicting Gradients: An optimization technique that resolves destructive interference between task gradients. When two task gradients point in opposing directions, PCGrad projects each gradient onto the normal plane of the other.

• Preserves beneficial gradient components
• Eliminates destructive interference without task weighting
• Demonstrated to improve multi-task genomic models with heterogeneous assay targets