Soda Core

Soda Core uses a simple, human-readable YAML syntax (called a Soda Checks file) to define data quality tests. This declarative approach separates the what (the quality rule) from the how (the execution engine).

• Define checks for a dataset in a checks.yml file.
• Specify the data source (e.g., Snowflake, PostgreSQL, Spark).
• Write checks like row_count > 0 or missing_count(column_name) = 0.
• This configuration-as-code approach enables version control, peer review, and easy integration into CI/CD pipelines.

While checks are defined in YAML, they are executed via a Python library (soda-core-sql or soda-core-spark). This provides flexibility and power for advanced use cases.

• Install the relevant soda-core package for your data source.
• Run checks from a CLI command: soda scan -d your_data_source -c configuration.yml checks.yml.
• Integrate scans directly into Python data pipelines (e.g., Airflow DAGs, Databricks notebooks).
• The Python SDK allows for dynamic check generation and programmatic result handling.

The framework provides a library of standard metrics for common data quality dimensions, and allows for the creation of custom SQL metrics.

Built-in Metrics:

• Freshness: freshness - Time since the most recent data update.
• Volume: row_count, missing_count, missing_percentage.
• Schema: schema - Detects changes in column names, types, or order.
• Uniqueness: duplicate_count, distinct_count.
• Validity: invalid_count, invalid_percentage (using regex or format checks).

Custom SQL Metrics:

• Define your own SQL queries to calculate business-specific metrics (e.g., total_revenue, customer_churn_rate).
• Apply thresholds and checks directly to these custom metric results.

Soda Core includes automated monitoring for numeric metrics using statistical methods, moving beyond static thresholds.

• For metrics like row_count or avg_value, you can configure checks to use dynamic thresholds.
• The system learns from historical metric values to establish a normal range.
• It flags anomalies when current values deviate significantly from the learned pattern (e.g., using techniques similar to statistical process control).
• This is crucial for detecting data drift—gradual changes in data distributions that static thresholds would miss.

Soda Core is designed to be embedded at key stages of the data lifecycle, acting as a quality gate.

• Pre-Production (CI/CD): Run checks on pull requests to validate changes to data models before merging.
• Pipeline Execution: Integrate a soda scan as a task within an orchestration tool (e.g., Airflow, Dagster, Prefect). If checks fail, the pipeline can be configured to halt, preventing bad data from propagating.
• Production Monitoring: Schedule regular scans (e.g., hourly, daily) on live data assets to monitor ongoing health and trigger alerts.
• This enables continuous testing for data, mirroring software engineering best practices.

A declarative or programmatic statement that defines a condition data must satisfy to be considered valid. These rules form the building blocks of automated testing.

• Examples: A column must be non-null, values must fall within a numeric range, or a string must match a specific regex pattern.
• In frameworks like Soda Core, these are defined in YAML configuration files as 'checks'.
• They can be simple (schema validation) or complex (custom SQL queries).

A programmatic check within a data pipeline that verifies a specific condition about the data and raises an error or warning if false. It is the functional execution of a data quality rule.

• Key Mechanism: Acts as a logical gate; if the assertion passes, the pipeline proceeds. If it fails, the pipeline can halt or trigger an alert.
• Difference from a Rule: A rule is the definition; an assertion is the runtime execution of that definition.
• Commonly implemented in code (e.g., Python's assert statement) or within testing frameworks.

A formal agreement, often codified as code, that specifies the expected schema, data types, freshness, and quality guarantees for a data product.

• Purpose: Establishes a clear, enforceable interface between data producers and consumers, reducing pipeline breakage.
• Components: Typically includes schema definition, SLAs for freshness, and a set of required data quality rules.
• Relation to Testing: Tools like Soda Core can be used to operationalize and continuously validate the rules specified within a data contract.

A data quality test whose failure prevents a data pipeline from proceeding to the next stage, acting as a mandatory quality gate.

• Critical Function: Blocks bad data from propagating downstream, protecting analytics and machine learning models.
• Implementation: Often configured within orchestration tools (e.g., Apache Airflow, Dagster) or testing frameworks where a test failure raises a fatal error.
• Example: A test checking for a minimum row count in a daily ingestion table; if it fails, the dependent aggregation jobs are not triggered.