Genome in a Bottle (GIAB)

GIAB develops gold-standard variant call sets through the integration of multiple sequencing technologies, expert manual curation, and arbitration of discordant calls. These truth sets define high-confidence regions where variant calls are considered definitively known, enabling precise calculation of sensitivity and precision for benchmarking. The consortium uses a tiered confidence system to stratify regions by the certainty of the reference call, ensuring evaluators understand where performance metrics are most reliable.

GIAB defines a rigorous stratified benchmarking framework that evaluates variant caller performance across different genomic contexts rather than reporting a single aggregate metric. The methodology includes:

• Stratification by repeat content: Evaluating performance in simple repeats, segmental duplications, and low-complexity regions
• Stratification by GC content: Assessing accuracy across varying nucleotide compositions
• Stratification by variant type: Separate evaluation for SNPs, indels, and structural variants
• Stratification by allele frequency: Performance analysis for heterozygous vs. homozygous variants This granular approach reveals context-specific failure modes that would be masked by genome-wide averages.

GIAB has characterized a set of well-studied reference genomes from immortalized cell lines that are publicly available for physical and computational benchmarking. The flagship genomes include the Ashkenazi trio (son HG002, father HG003, mother HG004), which enables validation of Mendelian inheritance consistency, and the Han Chinese trio (HG005, HG006, HG007). These cell lines are distributed by the Coriell Institute and can be sequenced by any laboratory to generate their own benchmarking data, allowing direct comparison of pipeline performance against the published truth sets.

Unlike haploid reference genomes, GIAB truth sets provide fully diploid genotype calls that distinguish between heterozygous and homozygous variants. For the HG002 genome, the consortium has produced a telomere-to-telomere diploid assembly using long-read sequencing, resolving both parental haplotypes across nearly all chromosomes. This phased assembly enables benchmarking of haplotype-aware variant callers and validation of phasing accuracy. The diploid nature of the truth set is critical for evaluating clinical pipelines where compound heterozygosity and variant phase can determine pathogenicity.

A machine learning technique that uses a Gaussian mixture model to assign a well-calibrated probability of error to each variant call. VQSR relies on high-confidence truth sets—such as GIAB's gold-standard calls—to train the model.

• Uses multiple annotation features like strand bias and mapping quality
• Distinguishes true variants from systematic sequencing artifacts
• Produces a Variant Quality Score Log Odds (VQSLOD) for filtering

A graphical plot illustrating the trade-off between sensitivity (recall) and positive predictive value (precision) of a variant caller across different confidence thresholds. GIAB's benchmark calls serve as the definitive ground truth for computing these curves.

• Stratified by variant type: SNPs vs. indels
• Stratified by genomic context: high-confidence regions vs. difficult-to-map regions
• Enables direct comparison between DeepVariant, GATK, and other callers

Statistical procedures applied to variant calling results to limit the expected proportion of false positives among declared discoveries. GIAB's high-confidence call sets define the boundaries of genomic regions where truth is known, enabling accurate FDR estimation.

• Benjamini-Hochberg procedure for multiple testing correction
• FDR thresholds typically set at 1% or 5% for clinical applications
• Critical for maintaining specificity in diagnostic pipelines

GIAB provides companion BED files that partition the reference genome into regions of varying difficulty. These stratifications allow developers to benchmark variant calling performance in specific genomic contexts.

• Low-complexity regions: Simple repeats and homopolymers
• Segmental duplications: Paralogous sequences prone to mismapping
• Major Histocompatibility Complex (MHC): Highly polymorphic region
• All difficult regions: Union of all challenging contexts