Parquet Columnar Storage

Genomic variant call format (VCF) data is inherently sparse; most positions are reference homozygous. Parquet stores data by column, allowing run-length encoding (RLE) and dictionary encoding to compress millions of identical '0/0' genotypes into a few bytes. This achieves 10x-50x compression ratios compared to row-based TSV or VCF files, directly reducing cloud storage and egress costs for population-scale datasets like UK Biobank or gnomAD.

Parquet's internal metadata contains column chunk statistics (min/max values, null counts). When querying for rare variants with an allele frequency below 0.01, the query engine reads these statistics first and skips entire row groups where the max frequency is below the threshold. This predicate pushdown eliminates scanning terabytes of irrelevant data, reducing query times from hours to seconds for targeted variant analysis.

Genomic data is not flat. A single variant record contains nested structures: multiple alleles, per-sample genotype likelihoods (PL), and INFO field annotations. Parquet's Dremel encoding natively supports complex nested schemas with lists and structs, preserving the hierarchical relationships of FORMAT and SAMPLE columns without expensive JOIN operations or manual denormalization.

Modern query engines like Apache Arrow and DuckDB process Parquet data in columnar batches that map directly to CPU cache lines and SIMD instructions. When calculating population allele frequencies across a cohort, the engine loads a contiguous array of allele counts and executes a single vectorized division operation, achieving near-memory-bandwidth throughput that is orders of magnitude faster than row-at-a-time processing.

Genomic annotation databases continuously add new fields—CADD scores, SpliceAI predictions, ClinVar classifications. Parquet supports additive schema evolution, allowing new columns to be appended without rewriting existing data or breaking downstream consumers. This enables bioinformatics pipelines to incrementally enrich variant datasets without costly full-dataset migrations.

Parquet integrates with Hive-style partitioning to organize data by chromosome or genomic interval. A query targeting only chromosome 21 can prune all partitions for chromosomes 1-20 and 22-XY by inspecting directory metadata, never opening those files. Combined with predicate pushdown, this two-level pruning enables interactive-speed queries on whole-genome datasets containing billions of variants.