Declarative Data Tests

Tests are defined as configuration artifacts (e.g., YAML, JSON, SQL snippets) rather than imperative code. This separates the validation logic from the execution engine, enabling:

• Portability: The same test definition can be executed across different runtime environments (Spark, Snowflake, BigQuery).
• Version Control: Test logic is stored in Git, enabling code review, rollback, and audit trails.
• Centralized Management: A single source of truth for all data quality rules across an organization.

The declarative model enforces a clean separation between the what (the validation rule) and the how (the execution engine). The test author specifies the expected condition (e.g., column X must be unique), and the underlying Data Quality Rule Engine determines the optimal execution path. This abstraction:

• Simplifies authoring for data practitioners who understand business logic but not distributed computing.
• Enables engine optimization, as the platform can choose the most efficient query or algorithm.
• Facilitates tool agnosticism, allowing organizations to switch execution backends without rewriting tests.

Declarative tests are built from atomic, reusable components. Common patterns include:

• Schema Validations: Checks for data type, nullability, and allowed values.
• Freshness Checks: Ensures data is updated within a specified time window.
• Volume Anomalies: Detects unexpected spikes or drops in row counts.
• Custom SQL Assertions: Allows for complex business logic validation. These components can be composed into test suites applied to multiple tables or datasets, ensuring consistent quality standards. Changes to a base rule propagate automatically to all dependent tests.

Declarative tests are designed to be executed as automated checkpoints within modern data workflows:

• CI/CD for Data: Tests run automatically in pre-merge checks for schema changes or new data pipeline code.
• Data Quality Gates: Tests act as gates in orchestration tools (e.g., Airflow, Dagster), preventing downstream processes from consuming invalid data.
• Shift-Left Testing: Quality rules are applied to development and staging environments, catching issues before they reach production. This practice is foundational to Data Reliability Engineering (DRE).

This contrasts with imperative data testing, where validation logic is written as procedural code (e.g., Python scripts with explicit loops and conditionals).

Declarative (YAML Example):

yamlCopy
test_type: uniqueness
table: orders
columns: [order_id]

Imperative (Python Pseudocode):

pythonCopy
def test_uniqueness():
    df = spark.table('orders')
    duplicate_count = df.groupBy('order_id').count().filter('count > 1').count()
    assert duplicate_count == 0

The declarative approach is more concise, less error-prone, and easier for non-engineers to audit and modify.

Declarative tests are used to enforce key dimensions of the Data Health Score:

• Accuracy: Data correctly reflects the real-world entity it represents (e.g., price > 0).
• Completeness: Mandatory fields are populated (e.g., customer_id IS NOT NULL).
• Consistency: Data conforms to uniform formats and units across systems.
• Timeliness/Freshness: Data is available within an expected timeframe (e.g., max(timestamp) >= NOW() - INTERVAL '1 hour').
• Uniqueness: No unintended duplicate records exist.
• Validity: Data adheres to defined business rules and domain constraints.

Ensures data adheres to expected structural formats and data types. This foundational test prevents downstream errors caused by malformed data.

• Column Existence: Verifies a required column is present in a table.
• Data Type Conformance: Checks that values in a column (e.g., customer_id) match the declared type (e.g., INTEGER, TIMESTAMP).
• Nullability Constraints: Enforces whether a column can contain NULL values.

Example YAML rule: assert_column_exists: table: orders, column: order_id

Monitors the timeliness of data updates to ensure information is current and meets Service Level Objectives (SLOs) for delivery.

• Maximum Allowable Delay: Asserts that the most recent record in a table is no older than a threshold (e.g., updated_at >= NOW() - INTERVAL '1 hour').
• Ingestion Heartbeat: Validates that a pipeline has successfully written data within a scheduled window.

These tests are critical for time-sensitive use cases like dashboards and real-time analytics, directly impacting data downtime metrics.

Validates that data arrives in expected quantities and is not missing critical segments.

• Row Count Boundaries: Declares that a table should have between an expected minimum and maximum number of rows after a refresh (e.g., row_count between 1000 and 5000).
• Non-Null Completeness: Ensures a critical column has no missing values (e.g., non_null_count(email) = total_row_count).
• Uniqueness: Asserts that values in a key column are distinct (e.g., unique_count(order_id) = total_row_count).

These tests catch pipeline breaks, incomplete loads, and duplicate generation issues.

Enforces consistency and valid relationships between different datasets, a core principle of data reliability engineering.

• Foreign Key Validity: Verifies that all values in a child table's column (e.g., orders.customer_id) exist in the parent table (customers.id).
• Orphan Record Detection: Identifies records in a reference table that are no longer linked to any primary record.

Declarative SQL example: SELECT COUNT(*) FROM orders o LEFT JOIN customers c ON o.customer_id = c.id WHERE c.id IS NULL must equal 0.

Uses statistical methods to validate that data values fall within expected numerical ranges and distributions, often leveraging dynamic baselines.

• Value Range: Asserts that a numeric column's values (e.g., transaction_amount) stay within a sane min/max (e.g., 0 <= value <= 1000000).
• Average/Median Stability: Checks that aggregate metrics do not deviate beyond a percentage threshold from a rolling historical average.
• Standard Deviation Bound: Flags if the dispersion of a column's values exceeds an expected limit, which can indicate data corruption.

These tests are proactive guards against data drift and subtle quality degradation.

Encodes domain-specific rules that data must satisfy to be considered accurate for business operations.

• Calculated Field Accuracy: Validates that a derived column (e.g., total_price) correctly equals unit_price * quantity - discount for all rows.
• State Transition Validity: Ensures status fields follow a permitted sequence (e.g., an order cannot move from shipped back to pending).
• Cross-Table Consistency: Asserts that aggregated metrics in a summary table match the sum of granular records in a source table.

These declarative rules move critical business knowledge out of application code and into observable, version-controlled data monitoring as code.

A Data Quality Rule Engine is the software component that executes the logic defined in Declarative Data Tests. It parses the high-level configuration (e.g., YAML, SQL), translates it into executable validation code, runs the checks against the target data, and produces pass/fail results. This engine is the runtime that brings declarative tests to life, often integrated directly into data pipelines or orchestration tools.

Data Monitoring as Code is the overarching practice that Declarative Data Tests exemplify. It involves defining all data quality checks, observability rules, and alerting logic in version-controlled, machine-readable configuration files (like YAML). This enables:

• Versioning and peer review of quality rules.
• Automated deployment of monitoring alongside pipeline code.
• Consistency and reproducibility across development, staging, and production environments.
• Infrastructure-as-Code (IaC) principles applied to data reliability.

CI/CD for Data is the pipeline automation framework into which Declarative Data Tests are integrated. It applies software engineering's Continuous Integration and Continuous Delivery practices to data assets. Declarative tests act as quality gates within this pipeline:

• Continuous Integration: Tests run automatically when new data pipeline code or schema definitions are committed.
• Continuous Delivery: Tests validate data quality in staging environments before promoting datasets to production.
• Rollback Triggers: Failed tests can automatically prevent faulty data from being published or trigger rollbacks.

A Data Quality Gate is a specific checkpoint, often enforced by Declarative Data Tests, that blocks the progression of a data pipeline unless validation rules are satisfied. It is the practical implementation of a test in a workflow. For example:

• A gate after a raw data ingestion job that checks for NOT NULL constraints on key columns.
• A gate before a BI dashboard updates that validates row counts are within expected bounds.
• Gates ensure that only data meeting a defined quality threshold proceeds to downstream consumers, preventing "bad data" from propagating.

Data Contract Monitoring uses Declarative Data Tests as its enforcement mechanism. A data contract is a formal agreement between a data producer and consumer specifying schema, semantics, freshness, and quality guarantees. Declarative tests codify these guarantees:

• Schema Validation: Tests enforce column names, data types, and allowed values.
• Semantic Rules: Tests validate business logic (e.g., revenue = quantity * price).
• SLI/SLO Compliance: Tests measure indicators like freshness or completeness against agreed objectives.
• Automated monitoring of these tests ensures contractual compliance is continuously verified.

Automated Data Profiling is a complementary discovery process that often informs the creation of Declarative Data Tests. Profiling tools automatically analyze a dataset to infer its structure, statistics, and patterns. The outputs—such as discovered distributions, uniqueness, and null rates—provide the empirical basis for writing meaningful declarative rules. For instance, profiling might reveal that a customer_id column is 100% unique, leading to the creation of a declarative test asserting column_is_unique(customer_id).