Cooler File Format

Cooler stores sparse genomic interaction matrices in an HDF5 container using a chunked, columnar layout. This design allows for efficient random access to arbitrary sub-matrices without loading the entire dataset into memory. The chunking strategy partitions the genome into fixed-size bins, enabling out-of-core computation for datasets that exceed available RAM. Each chunk is independently compressed, balancing I/O throughput with storage efficiency.

Cooler supports multi-resolution contact matrices through a recursive pixel aggregation scheme. Starting from a base resolution (e.g., 1 kb), coarser resolutions are generated by summing contacts within larger bins. This hierarchical structure enables zoomable analysis, where users can query low-resolution genome-wide views or high-resolution locus-specific interactions. The aggregation preserves the sparse representation at each level, avoiding the storage explosion of dense matrices.

The format provides an API for range queries that retrieve contact frequencies between two genomic intervals. Using the matrix() method, users can extract sub-matrices by specifying chromosome coordinates and resolution. Strided access allows sampling every nth bin, useful for downsampling or generating training batches for deep learning models. The query engine leverages the chunk index to minimize disk reads, achieving sub-second retrieval for targeted regions.

Cooler files are anchored to a reference genome assembly via an embedded chromosome table. This table maps chromosome names to their lengths and ordering, ensuring that matrix coordinates correspond to actual genomic positions. The anchoring enables cross-dataset comparisons by aligning contact maps from different experiments to the same coordinate system. It also supports assembly-aware normalization, where bin-level biases are stored alongside the interaction data.

Cooler stores both raw contact counts and normalized interaction frequencies in separate matrices within the same file. Normalization vectors, such as iterative correction weights or distance-expected values, are stored as 1D arrays alongside the 2D pixel data. This design allows users to toggle between raw and normalized views without recomputing corrections. The balanced matrix representation is critical for training deep learning models that require bias-corrected input features.

The format supports streaming access for sequential processing of entire chromosomes, enabling memory-efficient training loops. Pixel data can be iterated in genomic order or matrix order, depending on the downstream algorithm. Cooler's HDF5 backend allows concurrent read access from multiple processes, facilitating parallel data loading in distributed training setups. Write operations are serialized to maintain data integrity during matrix construction.