Multi-Scale Embedding

The architecture employs multiple convolutional kernels of varying sizes (e.g., 3, 9, 27 nucleotides) operating on the same input sequence simultaneously. Each kernel size captures a distinct biological scale:

• Small kernels (1-3 bp): Capture dinucleotide biases, splice junctions, and point mutations
• Medium kernels (9-27 bp): Detect transcription factor binding motifs and codon usage patterns
• Large kernels (81-243 bp): Model nucleosome positioning and local chromatin structure The outputs are concatenated into a multi-resolution feature map before downstream processing.

Instead of parallel branches, some architectures extract multi-scale features through successive transformer layers with progressive pooling operations. Each layer captures a different level of abstraction:

• Early layers: Learn local nucleotide dependencies and short-range interactions
• Middle layers: Aggregate into motif-level and exon-level representations
• Deep layers: Encode gene-level and long-range enhancer-promoter interactions This mimics the hierarchical organization of the genome itself, from base pairs to topologically associating domains (TADs).

Borrowed from computer vision architectures like FPN (Feature Pyramid Networks), this mechanism fuses feature maps from different scales through top-down pathways and lateral connections. In genomics:

• Bottom-up pathway: Standard feedforward computation at increasing coarseness
• Top-down pathway: Upsampled coarse features are merged with fine-grained features
• Lateral connections: 1x1 convolutions align channel dimensions before element-wise addition This ensures that high-resolution nucleotide-level information is enriched with broader genomic context.

Rather than manually specifying kernel sizes or pooling ratios, learned gating mechanisms allow the model to dynamically weight the contribution of each scale for a given genomic region. Implementation approaches include:

• Squeeze-and-excitation blocks: Recalibrate channel-wise feature responses based on global context
• Gated linear units (GLU): Learn to suppress irrelevant scales for specific prediction tasks
• Cross-scale attention: Allow features at one resolution to directly attend to features at another This enables the model to automatically emphasize motif-level features in regulatory regions while prioritizing domain-level features in gene deserts.

Training signals are applied at multiple levels of the architecture, not just the final output. This technique, known as deep supervision, ensures that intermediate representations learn meaningful biological features:

• Auxiliary classifiers attached to intermediate layers predict the same target from partial features
• Contrastive losses at each scale ensure that representations of functionally similar elements cluster together
• Reconstruction losses on pooled representations preserve fine-grained information through the hierarchy This prevents vanishing gradients and forces each scale to capture biologically interpretable patterns.

An alternative to parallel branches uses stacked dilated convolutions with exponentially increasing dilation rates (1, 2, 4, 8, 16...). This achieves an exponentially growing receptive field without losing resolution or increasing parameter count:

• Dilation rate 1: Standard convolution, captures adjacent nucleotide dependencies
• Dilation rate 4: Skips 3 positions, captures codon-level patterns
• Dilation rate 16: Skips 15 positions, captures enhancer-level interactions This approach, popularized by WaveNet and adapted for genomics, enables processing of 100k+ base pair sequences with a logarithmic number of layers.

A family of transformer modifications that reduce the quadratic memory complexity of self-attention by restricting each token to attend only to a subset of other tokens. Key implementations include:

• BigBird: Combines random, local, and global attention patterns
• Longformer: Uses a sliding window with dilated attention
• Sparse Transformer: Applies strided attention patterns

These mechanisms enable processing of full-length genomic loci without truncation, preserving multi-scale context from individual nucleotides to entire gene neighborhoods.

A tokenization strategy that segments an mRNA transcript or coding DNA sequence into non-overlapping triplets of nucleotides, directly aligning the vocabulary with the fundamental functional units of the genetic code. This approach creates a natural mid-scale representation between single nucleotides and full genes, allowing models to learn codon-level patterns such as synonymous substitution preferences and translational efficiency signals without explicit feature engineering.

A self-supervised representation learning method that trains an encoder to extract slowly varying features by maximizing mutual information between latent representations of distant sequence patches. In genomics, CPC learns a hierarchy where:

• Local features capture nucleotide-level motifs
• Global features encode chromatin domain properties

This naturally produces a multi-scale embedding space where different layers correspond to different biological resolutions.

A mechanism that injects information about the absolute or relative position of each token into the input embedding, enabling permutation-invariant transformer architectures to process sequential genomic data. Rotary Position Embedding (RoPE) is particularly relevant for multi-scale models as it encodes position with rotation matrices and naturally incorporates relative position dependency, allowing length extrapolation to sequences longer than those seen during training.

A special aggregation token prepended to the input sequence in BERT-based genomic models whose final hidden state serves as a fixed-length, pooled representation of the entire sequence. This token effectively learns to integrate signals from all scales—nucleotide, motif, and domain-level—into a single vector for downstream tasks like promoter strength prediction or enhancer classification, acting as a learned multi-scale summary.