Transfer Learning

Models first learn the statistical structure of the genome by solving pretext tasks on massive unlabeled datasets.

• Masked Language Modeling: Predicts masked nucleotides from sequence context (e.g., DNABERT).
• Next Token Prediction: Autoregressive modeling of genomic k-mers.
• This phase builds a foundation model that internalizes motifs, syntax, and long-range dependencies without requiring expensive experimental labels.

The pretrained model is adapted to a specific downstream task like gene expression prediction using a smaller labeled dataset.

• Only the final layers may be retrained (linear probing), or the entire network is updated with a low learning rate.
• This transfers learned regulatory grammar—enhancer logic, splice sites, and chromatin context—to the new task.
• Dramatically reduces the number of labeled examples required compared to training from scratch.

Representations learned from one epigenomic assay are reused to predict a different data modality.

• A model pretrained on ATAC-seq (chromatin accessibility) can be fine-tuned to predict RNA-seq expression.
• The shared latent space captures fundamental cis-regulatory logic that generalizes across molecular phenotypes.
• This is critical when the target modality has limited training samples but a related, abundant modality exists.

Genomic models pretrained on one organism can be transferred to another, exploiting evolutionary conservation.

• A model trained on the human genome (ENCODE, GTEx) can be fine-tuned on mouse or zebrafish regulatory data.
• Transfer works because core molecular mechanisms—transcription factor binding, splicing, chromatin remodeling—are deeply conserved.
• Enables functional genomics in non-model organisms where large-scale training data is unavailable.

Instead of a single pretext task, the model is simultaneously trained on multiple genomic prediction tasks across diverse cell types and assays.

• A shared trunk network learns universal genomic features while task-specific heads specialize.
• This acts as a powerful regularizer, preventing overfitting to any single data distribution.
• The resulting representations are more robust and transferable to entirely novel, unseen tasks.

Transfer learning techniques are used to align models trained on one sequencing platform or laboratory protocol to perform accurately on another.

• Adversarial domain adaptation removes technical confounders while preserving biological signal.
• A model trained on GTEx data can be adapted to a clinical lab's RNA-seq pipeline without retraining from scratch.
• Critical for deploying research-grade models in clinical production environments with different data characteristics.

A training strategy where models learn intrinsic genomic patterns from unlabeled DNA sequences through pretext tasks like masked language modeling. By predicting masked nucleotides from surrounding context, models like DNABERT and the Nucleotide Transformer develop rich, transferable representations of regulatory grammar, chromatin organization, and evolutionary constraints without requiring expensive experimental labels. These pretrained weights serve as the initialization point for fine-tuning on gene expression prediction, dramatically reducing the labeled data required.

A training paradigm where a single neural network is simultaneously trained on multiple related prediction tasks, such as forecasting expression across different tissues, cell types, or species. By sharing hidden representations, the model learns generalizable regulatory logic that transfers across contexts. For gene expression, multi-task architectures often predict CAGE-seq, RNA-seq, and PRO-seq signals jointly, leveraging correlations between transcript initiation, elongation, and steady-state abundance to improve performance on each individual task.

Techniques that bridge the distribution gap between the source domain where a model was pretrained and the target domain of expression prediction. Common genomic domain shifts include:

• Species transfer: Adapting human-trained models to mouse or zebrafish genomes
• Assay transfer: Converting chromatin accessibility models to expression predictors
• Tissue transfer: Generalizing from well-characterized tissues to rare cell types Methods include adversarial domain alignment, where a discriminator network forces the feature extractor to produce domain-invariant representations.

During fine-tuning, earlier layers that encode universal sequence motifs such as transcription factor binding sites and splice junctions are largely preserved, while later layers reorganize to predict expression levels. Catastrophic forgetting occurs when aggressive fine-tuning overwrites useful pretrained features. Mitigation strategies include:

• Elastic weight consolidation: Penalizing changes to parameters important for the original task
• Progressive networks: Freezing pretrained columns and learning lateral connections to new task-specific columns
• Experience replay: Interleaving pretraining objective examples during fine-tuning

The ability of genomic foundation models to predict expression in unseen cell types or conditions without any task-specific training examples. Zero-shot transfer relies entirely on the model's pretrained understanding of regulatory grammar. Few-shot transfer uses a minimal number of labeled examples, often 10-100, to rapidly adapt. The Nucleotide Transformer demonstrates this by generating embeddings that cluster by functional element type without explicit supervision, enabling expression prediction in rare tissues where large-scale RNA-seq data is unavailable.