Test Coverage (Data)

Measures the proportion of a dataset's structural elements validated by tests. This includes:

• Column-level validation: Tests for data types, nullability constraints, and allowed value enumerations.
• Table-level validation: Checks for primary key uniqueness, referential integrity between tables, and adherence to expected schemas.
• Example: A table with 20 columns where tests exist for data type and null constraints on all columns has 100% schema coverage for those constraints.

Assesses the extent to which complex, domain-specific rules are programmatically validated. This goes beyond basic schema checks to enforce business invariants.

• Key areas: Validating calculated fields (e.g., revenue = price * quantity), enforcing relationship rules (e.g., order_date <= shipment_date), and ensuring categorical mappings are correct.
• Critical metric: The percentage of documented business rules that have corresponding automated tests. Uncovered rules represent a direct business risk.

Evaluates if tests monitor the timeliness and completeness of data delivery, which are critical for operational dashboards and real-time models.

• Freshness Tests: Validate that data arrives within a Service Level Agreement (SLA), e.g., "The nightly sales table must be updated by 06:00 UTC."
• Volume Tests: Check for significant deviations in row counts or aggregate sums (e.g., daily transaction count is within ±10% of the 30-day moving average).
• Coverage Gap: A pipeline may have perfect schema tests but zero freshness checks, leading to stale data going undetected.

Measures the validation of data integrity across transformation steps and pipeline dependencies. It answers: "Are tests applied at every critical stage?"

• Upstream/Downstream Coverage: Tests on source data, intermediate tables, and final consumer-facing data products.
• Integration Test Coverage: Validates that joins, aggregations, and business logic produce correct results across multiple dependent tables.
• Impact: A failure in an upstream table with poor test coverage can cascade undetected through many downstream assets.

Tracks the validation of the statistical properties of data columns, which is essential for detecting data drift that can degrade machine learning model performance.

• Covered Metrics: Tests for mean, median, standard deviation, value distribution (histograms), and the presence of outliers.
• Dynamic Thresholds: Advanced coverage uses statistical process control to set automatic bounds, rather than static limits.
• Example: A model feature column customer_age has coverage if tests monitor for shifts in its average value or the introduction of invalid negative numbers.

The practice of quantitatively tracking test coverage metrics to identify and prioritize gaps. Effective measurement requires:

• A Coverage Registry: A catalog of all data assets (tables, columns) and business rules mapped to their corresponding tests.
• Gap Analysis: Identifying critical columns or rules with no associated tests, which represent the highest risk areas.
• Industry Benchmark: While 100% coverage is often impractical, high-priority data (e.g., financial reporting tables) should aim for >95% coverage on key dimensions.

A declarative or programmatic statement that defines a condition data must satisfy to be considered valid. These are the atomic units that compose a test suite.

• Examples: A column must be non-null, values must fall within a defined range, a foreign key must reference an existing record.
• Implementation: Can be written in SQL, Python, or configuration languages like YAML within frameworks such as Great Expectations or dbt.

A curated collection of data quality rules or assertions that collectively define the expected properties and behavior of a specific dataset. It represents the formal specification against which test coverage is measured.

• Function: Groups related rules (e.g., all rules for a customer_orders table) into a single, executable unit.
• Coverage Metric: The proportion of an asset's columns and business logic covered by its expectation suite directly quantifies test coverage.

An approach where data quality tests are specified as configuration stating what the data condition should be, rather than how to check it programmatically. This paradigm is central to scalable coverage.

• Mechanism: Uses frameworks like dbt Tests or Soda Core where tests are defined in YAML or SQL configurations.
• Advantage: Separates the test definition from execution logic, making test suites easier to write, maintain, and audit, thereby encouraging higher coverage.

A data quality test whose failure prevents a data pipeline from proceeding to the next stage. These tests act as critical quality gates to enforce coverage policies.

• Purpose: Blocks the propagation of invalid data, making test failures actionable and blocking.
• Coverage Role: High-priority rules for data freshness, volume, and critical schema integrity are typically configured as gated tests to ensure core coverage is always enforced.

The practice of managing data quality rules, test suites, and their configurations using version-controlled code (e.g., SQL, Python, YAML). This is the enabling discipline for measurable test coverage.

• Benefits: Enables peer review, change tracking, CI/CD integration, and reproducibility of quality checks.
• Coverage Impact: Treating tests as code allows coverage metrics to be tracked alongside pipeline code, ensuring quality evolves with the data asset itself.

The process of testing data against complex, domain-specific logic that defines correct business relationships and calculations. This represents the highest-value layer of test coverage.

• Examples: Validating that a customer's lifetime value calculation matches the business definition; ensuring regional sales totals roll up correctly to global figures.
• Coverage Gap: Many basic test suites cover schema and statistics but lack deep business rule validation, representing a critical coverage area for data trust.