Smoke Test (Data)

A data smoke test is designed for speed and efficiency, typically executing in minutes, not hours. It targets a minimal subset of the most critical data assets and business rules to provide a rapid go/no-go signal. This contrasts with comprehensive regression or integration test suites.

• Scope: Tests only core tables, key joins, and essential aggregations.
• Objective: Answer the question: "Did the basic pipeline work?" not "Is every detail perfect?"
• Speed: Execution time is a primary constraint; tests must be lightweight to avoid blocking deployment workflows.

The smoke test serves as a mandatory quality gate within a Continuous Integration/Continuous Delivery (CI/CD) pipeline for data. A failure should halt the deployment process, preventing potentially broken data from propagating to downstream consumers and analytical models.

• Pipeline-Gated: Integrated directly into deployment orchestration tools (e.g., Apache Airflow, Dagster, GitHub Actions).
• Fail-Fast: Designed to catch show-stopping errors early, such as connection failures, catastrophic null rates, or broken primary keys.
• Blocking Action: A test failure triggers an alert and stops the promotion of the new pipeline version to higher environments.

While limited in scope, smoke tests enforce non-negotiable data integrity constraints. These are the checks that, if failed, would render the data product useless or dangerously misleading.

Common validations include:

• Schema Conformance: Verify expected columns, data types, and nullability constraints are present.
• Freshness: Confirm new data has arrived within the expected time window.
• Volume Sanity: Check that row counts are within a plausible range (not zero, not 1000x normal).
• Key Uniqueness: Validate primary keys or business keys are unique and non-null.
• Critical Business Logic: Test one or two fundamental calculations or aggregations.

A smoke test is not a replacement for comprehensive testing. It is the first layer in a testing pyramid strategy for data, designed to catch gross errors quickly so that more resource-intensive tests can be run with confidence.

Testing Pyramid Analogy:

• Smoke Tests (Top): Few, fast, broad-stroke checks run on every commit.
• Integration Tests (Middle): Validate interactions between multiple pipeline components.
• Unit Tests (Base): Many isolated tests for individual transformation functions or SQL models.
• Regression Tests: Ensure new changes don't break existing functionality.

The smoke test's pass is a prerequisite for triggering these deeper, more expensive test suites.

Modern data smoke tests are defined and managed as code, following the Data Quality as Code principle. This allows for version control, peer review, reuse, and seamless integration into automated pipelines.

Implementation Frameworks:

• Great Expectations: Define an Expectation Suite with critical expectations and run it via a Checkpoint.
• dbt: Use core dbt test functionality on key models with severity: error.
• Soda Core: Write critical checks in a soda.yml file for freshness, row count, and uniqueness.
• Custom Scripts: Python or SQL scripts that run essential queries and assert results.

This codification ensures tests are reproducible, transparent, and part of the engineering workflow.

Smoke test execution is triggered by specific deployment events, not run on an arbitrary schedule. Its context is the validation of a change to the data system.

Common Trigger Events:

• Post-Build/Deployment: Immediately after a new version of a pipeline job or data model is deployed to a staging or production environment.
• On Data Arrival: For ingestion pipelines, run after a new batch of source data is landed but before any complex transformations begin.
• Schema Migration: Following an ALTER TABLE operation or similar structural change to a database.

The test uses the newly deployed code and the current production data (or a recent sample) to validate the change in context.

Verifies that the incoming data matches the expected schema, including column names, data types, and nullability constraints. A failure here indicates a breaking change at the source or a misconfigured ingestion process.

• Key Checks: Column presence, data type (e.g., INTEGER, TIMESTAMP), and nullability.
• Example: A pipeline expecting a customer_id column of type VARCHAR fails if the source suddenly sends it as INTEGER or renames it to client_id.
• Impact: Downstream joins, aggregations, and type-specific transformations will fail.

Confirms that new data has arrived within the expected time window. This test guards against silent failures where pipelines stall but don't crash.

• Key Checks: Maximum age of the most recent record or partition.
• Example: A daily sales table must have a partition for today's date by 07:00 UTC. The smoke test fails if the latest partition is more than 24 hours old.
• Tools: This is often implemented by checking MAX(timestamp) or metadata from systems like Apache Airflow.

Validates that the data volume is within an expected range, indicating a complete data transfer and guarding against partial loads or source system outages.

• Key Checks: Row count thresholds (minimum/maximum) or completeness percentage.
• Example: A nightly user event feed typically contains 1M ± 100k records. A smoke test fails if the count is 50k (suggesting a partial dump) or 0 (total failure).
• Dynamic Thresholds: For highly variable data, bounds can be set as a percentage of a rolling historical average.

Ensures that key identifier columns and critical business fields contain no null values, preserving referential integrity and analytical validity.

• Key Checks: Null count for primary keys, foreign keys, and essential dimensions.
• Example: In an orders table, order_id and customer_id must be 100% non-null. A single null in these columns can break downstream joins and aggregations.
• Scope: This is not a full data profiling exercise; it targets only the 3-5 most vital columns.

Asserts that the primary key column(s) for a table are unique, preventing duplicate records that can skew analytics and cause incorrect aggregations.

• Key Checks: Duplicate count for the defined primary key.
• Example: For a users table with user_id as the primary key, the test validates COUNT(DISTINCT user_id) = COUNT(*).
• Business Impact: Duplicate keys lead to double-counting in financial reports and corrupted machine learning features.

Verifies that values in critical enumerated columns fall within a known set of acceptable categories or ranges.

• Key Checks: Membership in a predefined list or within a numeric range.
• Example: A status column must only contain values from the set ('active', 'inactive', 'pending'). A value of 'cancelled' would cause a failure.
• Application: Often used for country codes, transaction types, or pipeline execution states to catch unexpected enum expansions.

A Data Quality Gate is a predefined quality threshold or set of passing tests that must be met for data to progress from one stage of a pipeline to another or to be deemed fit for consumption. It acts as a mandatory checkpoint.

• Pipeline-Gated Tests are a common implementation, where a test failure blocks the pipeline.
• Gates can be based on schema validation, statistical thresholds, or business rule checks.
• This concept enforces quality as code by programmatically preventing bad data from contaminating downstream models and dashboards.

An Assertion is a programmatic check within a data pipeline that verifies a specific condition about the data and raises an error or warning if the condition is false. It is the atomic unit of a data test.

• Examples include checking for non-null values, primary key uniqueness, or that a summed column matches a known total.
• Assertions are the building blocks for smoke tests and more comprehensive expectation suites.
• Frameworks like Great Expectations and dbt provide structured ways to define and execute assertions.

A Pipeline-Gated Test is a data quality test whose failure prevents a data pipeline from proceeding to the next stage. It acts as an automated quality gate to block bad data from propagating downstream.

• This is a critical implementation of fail-fast principles in data engineering.
• Smoke tests are often configured as gated tests immediately after a deployment to catch catastrophic failures.
• Gating prevents data incidents and reduces the mean time to detection (MTTD) for pipeline issues.

Continuous Testing is the practice of automatically executing data quality tests at various stages of the data lifecycle to ensure ongoing data integrity. It integrates testing into the CI/CD pipeline.

• Tests run on pull requests, during pipeline execution, and in production monitoring.
• It encompasses smoke tests, regression tests, and canary tests.
• This practice is enabled by treating data quality as code and using test orchestration tools.

A Canary Test is a strategy where new data or a new pipeline transformation is released to a small, controlled subset of the production environment first. Quality tests validate its behavior before a full rollout.

• It is a risk mitigation technique for deploying high-impact changes.
• Smoke tests are often run as part of the canary validation to ensure core functionality works.
• If the canary tests pass, the change is deployed more broadly; if they fail, the rollout is halted with minimal impact.

Test in Production is the practice of running a subset of non-destructive data quality checks directly on live production data to monitor its health and catch issues that may not appear in pre-production environments.

• Smoke tests are a prime candidate for production execution after deployments.
• These tests must be non-breaking (e.g., read-only checks, sampling) to avoid impacting live services.
• It complements pre-production testing by validating data against real-world volume, distribution, and freshness.