Neural Pileup Representation

Encodes aligned reads as a 6-channel image-like tensor where each channel captures a distinct biological or technical feature:

• Channel 1 (Reference Base): One-hot encoded reference allele (A, C, G, T)
• Channel 2 (Read Base): Observed base from each aligned read
• Channel 3 (Base Quality): Phred-scaled quality score mapped to [0,1]
• Channel 4 (Mapping Quality): Confidence of correct read alignment
• Channel 5 (Strand Direction): Binary indicator for forward/reverse strand
• Channel 6 (Read Position): Normalized position within the read

This structured representation allows the CNN to learn complex interactions between sequence context and quality metadata simultaneously.

For each candidate variant locus, a fixed-size window (typically 100-221 base pairs) is extracted from the alignment pileup. The window is centered on the candidate position, with reads stacked vertically by alignment position. This creates a height × width × channels tensor where:

• Height = number of reads covering the locus
• Width = window size in base pairs
• Channels = feature dimensions

Padding and truncation strategies handle edge cases near chromosome boundaries and regions with extreme coverage depth. The fixed dimensions enable efficient batch processing across thousands of candidate loci.

Before tensor construction, reads are locally realigned around candidate variants to correct for misalignments caused by indels. This preprocessing step:

• Reduces false positive indel calls from alignment artifacts
• Ensures consistent read stacking in the pileup image
• Preserves phase information by maintaining read pair relationships

Advanced implementations incorporate haplotype-tagged channels that distinguish reads supporting different alleles, enabling the CNN to learn diploid genotype likelihoods directly from the tensor representation rather than relying on post-hoc statistical models.

Raw sequencing quality scores undergo base quality score recalibration (BQSR) before tensor encoding. The recalibrated scores are normalized and embedded as continuous-valued channels, allowing the CNN to:

• Learn to down-weight reads with systematically low quality
• Detect and compensate for strand bias artifacts
• Identify homopolymer indel errors through spatial patterns in quality degradation

This integration eliminates the need for separate hard-filtering steps and enables the model to make nuanced decisions based on the full quality landscape rather than thresholded binary filters.

The fixed-dimensional tensor format enables high-throughput inference by stacking thousands of candidate loci into mini-batches. Key optimizations include:

• Zero-padding reads to a uniform depth across loci
• Tensor memory layout optimized for GPU convolution operations
• Parallel processing of independent genomic regions

Production systems like DeepVariant process entire human genomes in under 5 hours on a single GPU, with the pileup tensor construction and CNN inference stages accounting for less than 30% of total runtime. This efficiency makes neural variant calling viable for population-scale studies.

The neural pileup representation naturally extends to joint calling scenarios by adding sample-specific channels or stacking tensors from multiple individuals. Approaches include:

• Population reference channels: Aggregate allele frequencies from external cohorts
• Trio-aware encoding: Parent-child read stacks for de novo mutation detection
• Tumor-normal paired channels: Separate channels for matched samples in somatic calling

These extensions allow the CNN to learn population-level priors and inheritance patterns directly from the tensor structure, improving sensitivity for rare variants while maintaining specificity against sequencing artifacts.

The specific process of converting a vertical stack of aligned reads at a candidate genomic locus into a multi-channel tensor or image. Each pixel row represents a read, columns represent genomic positions, and RGB-like channels encode base identities (A, C, G, T), quality scores, and strand direction. This transformation recasts a statistical genetics problem into a computer vision task solvable by convolutional neural networks.

A compact representation within SAM/BAM alignment files that summarizes the sequence of operations required to align a read to a reference genome. Operations include:

• M: Alignment match (can be sequence mismatch)
• I: Insertion in the read relative to reference
• D: Deletion from the reference
• S: Soft-clipped bases (not aligned but retained)

Neural pileup encoders parse CIGAR strings to correctly position bases and represent indel evidence in the tensor.

A critical preprocessing step that applies machine learning to correct systematic errors in per-base quality scores reported by sequencers. These errors arise from:

• Sequencing cycle position effects
• Dinucleotide context biases
• Flow cell lane variability

Recalibrated quality scores provide more accurate channel weights in the neural pileup tensor, directly improving variant calling precision.

A systematic sequencing error where a variant allele appears predominantly on reads from one DNA strand (forward or reverse). Neural pileup representations encode strand information as a separate tensor channel, allowing convolutional filters to learn patterns that distinguish true heterozygous variants (balanced strand representation) from technical artifacts (strand-skewed). This is a key advantage over traditional statistical callers.

An alternative to image-based pileup encoding that represents read alignments as a graph structure rather than a fixed grid. Nodes represent reads and reference positions; edges represent alignment relationships. Message-passing operations propagate information between connected reads and candidate alleles. This approach naturally handles variable read lengths and complex structural variants without the fixed-dimension constraints of tensor representations.