Declarative Testing (Data)

Tests are defined as declarative statements in structured files like YAML, JSON, or SQL, not as imperative scripts. This separates the specification of the rule (e.g., column X must be unique) from the underlying execution engine that runs the check. It enables non-engineers to read and contribute to test definitions and allows the testing framework to optimize execution.

The core principle is divorcing the test intent from the implementation details. A data engineer defines the business rule (the 'what'), and the testing framework handles the 'how'—generating the optimal query, managing database connections, and executing the validation. This clean abstraction makes test suites more portable and less brittle to underlying infrastructure changes.

Declarative rules are inherently modular and composable. A single rule validating an email format can be applied across dozens of tables without code duplication. Updates to a rule's logic are made in one configuration file, propagating the change everywhere it's referenced. This drastically reduces the technical debt associated with large-scale data quality initiatives.

Declarative tests are designed to plug into contemporary data workflows. They are natively supported by tools like dbt (.yml test blocks), Great Expectations (Expectation Suites), and Soda Core (checks.yml). This allows tests to be version-controlled alongside pipeline code, executed as pipeline-gated checks, and their results integrated into data observability platforms.

The syntax elevates testing from low-level schema checks to validating domain-specific business rules. Instead of just checking for non-null values, you can declare rules like:

• customer_lifetime_value must be recalculated daily.
• order_status transitions must follow a specific sequence.
• Regional sales_totals must reconcile with financial system reports. This shifts quality assurance from data integrity to business integrity.

Beyond static thresholds (e.g., value < 100), declarative frameworks support adaptive rules based on historical data. This includes:

• Statistical Process Control (SPC): Flag values that fall outside 3 standard deviations of a rolling mean.
• Seasonality-Aware Checks: Allow higher transaction volumes on Black Friday.
• Data Drift Detection: Automatically adjust thresholds as underlying data distributions evolve.

A declarative or programmatic statement that defines a condition data must satisfy to be considered valid. This is the fundamental building block of a test.

• Examples: 'Column X must be non-null', 'Values in Column Y must be between 0 and 100', 'This table's row count must match the source table's count'.
• Rules can be simple (schema validation) or complex (business logic). In declarative frameworks, these rules are expressed as configuration.

A programmatic check within a data pipeline that verifies a specific condition about the data and raises an error or warning if false. It is the execution mechanism for a data quality rule.

• Key Difference: While a 'rule' defines the what, an 'assertion' is the how—the actual code or operation that performs the validation.
• In declarative testing, the framework automatically generates and executes the underlying assertions from the declared rules.

A software methodology applied to data engineering where data quality tests are written before the pipeline code. This ensures the pipeline is built to satisfy explicit quality requirements from the start.

• Process: 1. Define expected data output and rules (declaratively). 2. Write failing tests. 3. Develop transformation code until tests pass.
• This practice embeds quality into the design phase, reducing defects and clarifying requirements for data producers and consumers.

The practice of managing data quality rules, tests, and configurations using version-controlled code (e.g., YAML, SQL, Python). This enables automation, peer review, and reproducibility, treating quality logic with the same rigor as application code.

• Benefits: Enables CI/CD for data, tracks historical changes to quality standards, and allows tests to be promoted through environments (dev, staging, prod) alongside pipeline code.
• Declarative testing frameworks like dbt and Soda Core are primary enablers of this paradigm.

A data quality test whose failure prevents the pipeline from proceeding to the next stage. It acts as a mandatory quality gate to block invalid or anomalous data from propagating downstream.

• Implementation: Often integrated into orchestration tools (e.g., Airflow, Dagster) where a task runs a validation suite; a failure triggers a pipeline failure and alert.
• This is critical for enforcing data contracts and protecting dependent models, dashboards, and applications from corruption.

A collection of data quality rules that define the expected properties of a specific dataset. This is a core concept in frameworks like Great Expectations.

• Structure: A suite groups related assertions (e.g., all checks for a customers table: uniqueness of customer_id, non-null email, valid signup_date range).
• Suites are typically stored as JSON or YAML and are executed as a unit by a checkpoint, providing a comprehensive validation profile for a data asset.