Assertion (Data)

This is the logical statement that defines what constitutes valid data. It is a Boolean expression that must evaluate to True for the data to pass.

• Examples: column_a is not null, column_b values are unique, revenue is greater than zero, customer_id matches a known pattern.
• The condition can be simple (single-column check) or complex (multi-column business logic).
• It is typically expressed in a declarative syntax (e.g., expect_column_values_to_be_between) or as a programmatic function (e.g., a Python assert statement).

This defines what data the assertion evaluates against. Precision in scope is critical for effective testing.

• Dataset: The specific table, file, or dataframe under test (e.g., prod.customers_table).
• Granularity: The level of data the check applies to:
  • Column-level: Assertions on a single column's values (e.g., uniqueness, nullness).
  • Row-level: Assertions that evaluate conditions across columns within a single row (e.g., end_date > start_date).
  • Table-level: Assertions about the dataset as a whole (e.g., row count, freshness timestamp).
  • Cross-table: Assertions about relationships between datasets (e.g., referential integrity).

This is the runtime system that executes the assertion condition against the target data. It handles the mechanics of data sampling, query execution, and condition evaluation.

• Frameworks: Tools like Great Expectations, dbt tests, or Soda Core provide built-in engines.
• Core Functions:
  • Query Generation: Translates the declarative rule into an efficient query (e.g., SQL, Spark).
  • Execution: Runs the query against the data source.
  • Result Calculation: Determines if the condition passed or failed, often computing a validation result (e.g., success: false, unexpected_count: 15).

This defines what happens when the assertion condition evaluates to False. It turns a check into a meaningful control point in a pipeline.

• Severity Levels:
  • ERROR / FAIL: The pipeline is halted, and an incident is raised. Used for critical business rules.
  • WARNING: The pipeline proceeds, but an alert is logged for investigation. Used for non-blocking anomalies.
• Actions: The engine can be configured to trigger specific workflows on failure:
  • Block the Pipeline: A pipeline-gated test that prevents downstream jobs from running.
  • Send Alerts: Notify teams via Slack, PagerDuty, or email.
  • Log to Metadata Store: Record the failure for audit trails and data incident management.

Every assertion execution produces a rich set of metadata, which is essential for observability and debugging. This is more than just a pass/fail boolean.

• Core Result: success (True/False).
• Quantitative Metrics: element_count, unexpected_count, unexpected_percent.
• Contextual Data:
  • Sample of Failing Rows: A subset of records that violated the condition.
  • Execution Timestamp: When the test ran.
  • Runtime Parameters: Any dynamic thresholds or variables used.
• This metadata is aggregated into test result dashboards and feeds data quality metrics and SLO calculations.

For numeric assertions, the threshold defines the acceptable bound for the metric being measured. It can be a fixed value or adapt to historical patterns.

• Static Threshold: A hard-coded, business-defined limit.
  • Example: row_count must be between 1000 and 1500.
• Dynamic Threshold: Calculated automatically based on statistical analysis of historical data, often using statistical process control.
  • Example: daily_order_volume must be within 3 standard deviations of the 30-day rolling average.
  • This is crucial for monitoring metrics with natural variance and enabling anomaly detection.

Applying software engineering testing methodologies to data workflows.

• Unit Tests (Data): Validate the logic of a single transformation (e.g., a SQL view, Python function) using a fixed, small fixture dataset. Frameworks like pytest are commonly used.
• Integration Tests (Data): Validate the correct interaction and data flow between multiple pipeline components (e.g., source → transformed table). They often use a test environment with sanitized or synthetic data. This approach ensures deterministic validation of transformation logic before data reaches production.

Data assertions follow specific operational patterns within a pipeline.

• Pipeline-Gated Tests: Assertions whose failure prevents the pipeline from proceeding, acting as a quality gate.
• Monitoring Assertions: Lighter-weight checks run in production to monitor health without blocking flow, often with dynamic thresholds.
• Test Orchestration: Automated scheduling and execution of test suites, managed by tools like Apache Airflow, Prefect, or Dagster.
• Result Aggregation: Collecting outcomes into dashboards (e.g., Grafana, Data Docs) for a unified view of data health.

Tools that perform deep comparisons between datasets, serving as a powerful form of assertion for change detection and validation.

• Purpose: Identify differences in row counts, column values, or schema between two data versions (e.g., production vs. staging, today vs. yesterday).
• Use Cases: Validating migration scripts, testing refactored pipelines, and ensuring data contract compliance.
• Examples: Open-source tools like data-diff or proprietary features within data platforms. This goes beyond simple counts to provide a granular delta analysis, crucial for regression testing.

A declarative or programmatic statement that defines a condition data must satisfy to be considered valid. This is the formal specification that a data assertion checks at runtime.

• Examples: A column must be non-null, values must fall within a predefined range, or a set of columns must be unique.
• Implementation: Can be expressed in SQL, YAML (e.g., dbt tests, Soda Core), or Python (e.g., Great Expectations).
• Purpose: Encodes business logic and integrity constraints, separating the 'what' from the 'how' of validation.

A formal agreement, often codified as code, that specifies the expected schema, data types, freshness, and quality guarantees for a data product. Data assertions operationalize the quality guarantees within a contract.

• Components: Includes service-level objectives (SLOs) for data, schema definitions, and explicit data quality rules.
• Function: Establishes a clear interface between data producers and consumers, reducing pipeline breakage.
• Analogy: Similar to an API contract, but for data streams and datasets.

In frameworks like Great Expectations, a collection of data quality rules or assertions that define the expected properties and behavior of a specific dataset. It is a reusable test suite for a data asset.

• Structure: Groups related data assertions (e.g., all checks for a customers table).
• Usage: Executed by a checkpoint to validate a batch of data.
• Benefit: Enables version-controlled, documented, and repeatable validation of entire datasets.

A data assertion or test whose failure prevents a data pipeline from proceeding to the next stage. It acts as a mandatory quality gate to block invalid data from propagating.

• Mechanism: Integrated into orchestration tools (e.g., Airflow, Dagster) to fail the pipeline task.
• Critical Checks: Validates primary key uniqueness, non-null critical columns, or row count thresholds.
• Outcome: Enforces fail-fast principles in data engineering, containing issues at the source.

A configured operation that runs a suite of data quality tests (an expectation suite) against a specific dataset and triggers actions based on validation results. It is the execution engine for assertions.

• Actions: Can update a data quality dashboard, send alerts, or pass/fail a pipeline-gated test.
• Configuration: Defines the data asset to test, the suite of assertions to run, and the result handlers.
• Automation: Enables continuous testing by being triggered on schedule or by pipeline events.

A software methodology applied to data engineering where data quality rules and assertions are written before the data pipeline code. The pipeline is then built to satisfy these explicit quality requirements.

• Process: 1. Define assertions for expected output. 2. Write pipeline code. 3. Run tests iteratively.
• Benefit: Ensures data quality is a design constraint, not an afterthought, leading to more reliable pipelines.
• Framework Support: Enabled by tools like Great Expectations and dbt that treat tests as first-class code.