Dilated Convolutions

The receptive field grows exponentially with each layer while the number of parameters remains constant. A standard 3x1 convolution with dilation rates of 1, 2, 4, 8, and 16 achieves a receptive field covering 31 nucleotide positions using only 15 trainable weights. This allows models like Basenji and Enformer to capture regulatory interactions spanning up to 200 kilobases without resorting to computationally prohibitive dense layers.

Unlike max-pooling or strided convolutions that downsample and discard spatial resolution, dilated convolutions preserve the output resolution of the feature map. This is critical for genomic tasks where single-nucleotide precision matters:

• No information loss: Every input position contributes to the output
• Constant parameter budget: A 3-wide kernel always has 3 weights regardless of dilation rate
• Memory efficiency: Models like SpliceAI use 32 dilated residual blocks to scan 10,000 nucleotides without exploding memory requirements

Stacking dilated convolutions with increasing dilation rates enables simultaneous capture of local motifs and distal regulatory elements. A typical genomic architecture processes:

• Short-range patterns (3-10 bp): Transcription factor binding motifs, splice donor/acceptor sites
• Mid-range interactions (100-1,000 bp): Nucleosome positioning, local chromatin structure
• Long-range dependencies (10-200 kb): Enhancer-promoter loops, topologically associating domains This hierarchical feature extraction mirrors the multi-scale nature of gene regulation.

When dilation rates share common factors (e.g., rates 2, 4, 8), the kernel samples input positions in a checkerboard pattern, leaving gaps that cause gridding artifacts. Modern genomic architectures address this through:

• Co-prime dilation rates: Using rates like 1, 2, 3, 5, 7 to ensure full coverage
• Dense-dilated stacking: Alternating standard convolutions with dilated layers
• Residual connections: Allowing gradient flow to bypass potentially sparse paths The Enformer architecture uses a combination of convolutional blocks and transformer attention to complement dilated receptive fields.

In autoregressive genomic models that predict nucleotides left-to-right, causal dilated convolutions ensure that the prediction at position i depends only on positions 0 through i-1. This is achieved by:

• Asymmetric padding: Zero-padding only the left side of the input
• Temporal ordering: Maintaining the sequential nature of DNA strands
• WaveNet-inspired architectures: Originally developed for audio generation, now adapted for DNA language models like DNABERT and the Nucleotide Transformer for masked nucleotide prediction tasks

Modern genomic architectures combine dilated convolutions with self-attention to capture both local sequence grammar and global context. Dilated convolutions excel at position-specific motif detection, while attention layers model content-dependent long-range interactions. The Enformer model demonstrates this hybrid approach:

• 7 convolutional blocks with dilation for local feature extraction
• 11 transformer layers for integrating distal enhancer-promoter interactions
• Output heads predicting 5,313 genomic tracks across human and mouse genomes This synergy achieves state-of-the-art performance on gene expression prediction from DNA sequence alone.

Dilated convolutions exponentially increase the receptive field without adding parameters or losing resolution. By inserting gaps (dilation rate d) between kernel elements, a 3×3 kernel with d=4 covers a 9×9 input area while using only 9 weights. This is critical for genomic models like Enformer and Basenji, which must capture regulatory interactions spanning up to 200 kilobases—enhancer-promoter loops, insulator boundaries, and long-range epistatic effects that standard convolutions would require dozens of layers to perceive.

The mathematical equivalent of dilated convolution, originating from the French term à trous ("with holes"). In signal processing, the à trous algorithm was developed for undecimated wavelet transforms. In deep learning, atrous convolutions enable dense feature extraction at multiple scales without downsampling—preserving spatial resolution while expanding context. This is essential for genomic base-pair-level prediction tasks where every nucleotide position requires a prediction, such as ChIP-seq peak calling or transcription factor binding site identification.

ASPP applies multiple dilated convolutions with different rates in parallel to capture multi-scale context simultaneously. A typical ASPP module uses rates like d=1, 6, 12, 18 to extract features at varying genomic distances—from local motif patterns to distal enhancer interactions. The outputs are concatenated and fused, giving downstream layers access to both fine-grained sequence features and broad regulatory context. This architecture is particularly effective for gene expression prediction where both proximal promoters and distal enhancers contribute to transcriptional output.

TCNs leverage dilated causal convolutions to model long-range sequential dependencies with parallel computation—unlike RNNs which process tokens sequentially. In genomics, TCNs applied to DNA sequences can process the entire 250 Mb of chromosome 1 in parallel, capturing dependencies across megabase-scale distances. The causal constraint ensures predictions at position i depend only on positions ≤ i, preserving the 5'→3' directionality of transcription. This architecture underpins models like SpliceAI, which predicts splice junctions from pre-mRNA context.

The systematic arrangement of dilation rates across layers determines the effective receptive field and the density of feature sampling. Common schedules include:

• Exponential: d=1, 2, 4, 8, 16... —maximizes receptive field growth
• Hybrid: repeating blocks of [1, 2, 4] —balances local and global features
• Prime-based: d=1, 2, 3, 5, 7... —avoids gridding artifacts where some input positions are never sampled

Poor scheduling creates gridding effects—systematic gaps in the receptive field that miss critical regulatory motifs.