Epigenomic Model Distillation

The foundational training paradigm where a compact student model is trained to mimic the output distribution of a larger, high-capacity teacher model. In epigenomics, the student learns to replicate the teacher's softmax probabilities over regulatory tracks rather than training directly on binary peak calls. The Kullback-Leibler divergence loss minimizes the difference between the teacher's softened logits and the student's predictions, transferring dark knowledge about inter-class similarities that one-hot labels miss.

The structural relationship between a pre-trained, frozen teacher network (e.g., Enformer or Basenji2) and a lightweight student network designed for inference efficiency. Key design choices include:

• Homogeneous distillation: Student shares the same architecture but with fewer layers or channels
• Heterogeneous distillation: Student uses a fundamentally different architecture (e.g., CNN student from a transformer teacher)
• Intermediate layer hints: The student is trained to match the teacher's hidden representations, not just final outputs, using L2 loss on feature maps

The objective function that drives the student to replicate the teacher's epigenomic track predictions across multiple cell types and assays. Unlike standard classification distillation, epigenomic distillation often uses a combination of:

• Mean squared error between student and teacher continuous-valued predictions
• Pearson correlation loss to preserve the shape of regulatory signal tracks
• Multi-task weighting to balance importance across DNase-seq, histone marks, and transcription factor binding profiles This ensures the student preserves the teacher's ability to distinguish active promoters from poised enhancers.

A hyperparameter T applied to the teacher's softmax that controls the softness of the probability distribution over epigenomic states. A higher temperature (e.g., T=4) flattens the distribution, revealing subtle inter-class relationships—such as the similarity between bivalent chromatin states and weak enhancers—that would be lost at T=1. The student is trained with the same high temperature but evaluated at T=1. This technique is critical for transferring the teacher's uncertainty about ambiguous regulatory regions.

A distillation strategy where the student learns to match the teacher's internal representations rather than just final predictions. In epigenomic models, this involves:

• Attention map transfer: The student's self-attention weights are trained to match the teacher's long-range enhancer-promoter interaction patterns
• Bottleneck layer alignment: The student's compressed latent space is regularized to preserve the teacher's manifold structure of cell-type-specific regulatory grammars
• Gram matrix matching: Preserving the pairwise similarity structure between genomic loci in the teacher's feature space

A specialized application where a distilled student model preserves the teacher's ability to perform in-silico mutagenesis and predict the functional impact of non-coding variants. The student must replicate:

• The teacher's predicted change in chromatin accessibility for reference vs. alternate alleles
• Allelic imbalance scores for heterozygous regulatory variants
• The ranking of pathogenic variants against benign background variation This enables clinical variant interpretation pipelines to run on CPU-only infrastructure without GPU dependency.