Data Quality Rule

Data quality rules exist on a spectrum from declarative to programmatic.

• Declarative Rules specify what condition the data must satisfy using configuration (e.g., YAML, SQL). Examples include column X must be non-null or values in column Y must be between 1 and 100. Frameworks like dbt tests and Soda Core use this approach.
• Programmatic Rules define how to check the condition using code (e.g., Python, Scala). This allows for complex, custom logic, such as validating business-specific calculations or multi-column relationships. Libraries like Great Expectations support both paradigms.

The choice depends on the complexity of the validation and the need for custom logic versus simplicity and maintainability.

Rules can be applied at different levels of a dataset, each serving a distinct purpose in the validation hierarchy.

• Schema-Level: Validate structural integrity, such as column existence, data types, and primary/foreign key constraints.
• Column-Level: Check properties of individual fields, including value ranges, allowed enumerations, format patterns (e.g., email, date), and nullability.
• Row-Level: Enforce conditions that apply to each record, like if column A = 'X', then column B must be > 10.
• Dataset-Level: Assess aggregate properties of the entire table or batch, such as row count thresholds, uniqueness of a composite key, or statistical summaries like mean and standard deviation.

Effective test coverage requires a mix of rules across all granularities.

A critical characteristic is whether a rule's pass/fail criteria are fixed or adaptive.

• Static Thresholds use hard-coded, absolute values (e.g., error_rate < 0.01%, latency < 500ms). They are simple to implement but can become brittle if underlying data patterns naturally evolve.
• Dynamic Thresholds are calculated automatically based on historical data, often using statistical process control (SPC) methods. For example, a rule might flag a value if it falls outside 3 standard deviations of the 30-day rolling mean. This adapts to seasonality and legitimate data drift, reducing false positives.

Dynamic thresholds are essential for monitoring metrics like daily row counts or aggregate sums in production environments.

Rules differ in their certainty of identifying an issue.

• Deterministic Rules produce a binary, unambiguous result. A value either conforms to a regex pattern or it does not; a foreign key either has a match in the parent table or it does not. These are used for enforcing absolute business logic and contractual guarantees.
• Probabilistic Rules identify anomalies or outliers based on statistical likelihood. They answer "is this data point unusual given historical context?" These are crucial for anomaly detection in metrics like transaction volumes or sensor readings, where some variation is expected, but significant deviations may indicate problems.

Most robust data quality postures employ a combination of both types.

Not all rule violations are equal. A core characteristic is defining the consequence of a failure.

• Severity Levels classify a rule's importance (e.g., BLOCKER, WARNING). A BLOCKER failure on a pipeline-gated test halts the data flow to prevent corrupt data from propagating. A WARNING may log an alert for investigation but allows the pipeline to proceed.
• Actionability dictates the response. Rules should be designed to trigger specific, useful actions:
  • Alert: Notify a team via Slack, PagerDuty, etc.
  • Quarantine: Route failing records to a holding area for manual review.
  • Block: Fail the pipeline job entirely.
  • Auto-remediate: Trigger a predefined corrective workflow.

Clear severity and action links are key to effective data incident management.

Modern data quality rules are not isolated checks; they are enriched with context and connected to the broader data ecosystem.

• Rule Metadata: Includes the rule's author, creation date, last modified timestamp, business owner, and associated SLAs or SLOs. This is vital for governance and accountability.
• Lineage Awareness: Rules should be linked to the specific data assets (tables, columns) they validate. When a rule fails, data lineage and dependency mapping tools can instantly visualize all downstream consumers impacted, drastically reducing triage time.
• Integration with Catalogs: Rule definitions and historical pass/fail results should be visible within enterprise metadata management and catalogs, providing a single pane of glass for data health.

This integration transforms rules from simple validators into intelligent components of a data observability platform.

These rules enforce the presence and basic correctness of data values. They are the most fundamental checks in any data quality suite.

• Non-null Rule: Ensures a required column contains no null or empty values (e.g., customer_id IS NOT NULL).
• Format Validity Rule: Validates that string data matches a defined pattern, such as an email address, phone number, or UUID regex.
• Domain Validity Rule: Checks that a value exists within a predefined set of allowed values (e.g., status IN ('active', 'inactive', 'pending')).

These rules verify that data correctly represents real-world entities and adheres to defined standards and schemas.

• Referential Integrity Rule: Ensures foreign key values in one table have a corresponding primary key in a related table, maintaining relational consistency.
• Schema Conformity Rule: Validates that a dataset's structure (column names, data types, nullability) matches a declared contract or expected schema.
• Cross-field Validation Rule: Checks logical relationships between columns (e.g., end_date must be after start_date, discount_amount cannot exceed total_amount).

These rules prevent data duplication and ensure uniform representation of information across the system.

• Primary Key Uniqueness Rule: Guarantees that a column or combination of columns meant to be a unique identifier contains no duplicate values.
• Business Key Uniqueness Rule: Ensures uniqueness on a domain-specific natural key (e.g., combination of user_email and signup_date).
• Consistency Rule: Validates that the same entity is represented identically across different tables or systems (e.g., a customer's name is spelled the same in all records).

These rules monitor the currency and delivery speed of data, which is critical for time-sensitive analytics and decision-making.

• Data Latency Rule: Measures the time elapsed between a real-world event and its availability in the data warehouse (e.g., event_ingestion_timestamp - event_occurred_timestamp < 5 minutes).
• Pipeline Freshness Rule: Checks that a table or data asset has been updated within an expected time window (e.g., MAX(updated_at) > CURRENT_TIMESTAMP - INTERVAL '1 hour').
• SLA/SLO Rule: A business-oriented rule defining the acceptable thresholds for data delivery, often tied to formal service-level agreements.

These rules use statistical methods to monitor the shape and properties of data, detecting anomalies and drift.

• Value Range Rule: Ensures numerical values fall within a minimum and maximum bound (e.g., age BETWEEN 0 AND 120). Can use dynamic thresholds based on historical percentiles.
• Statistical Distribution Rule: Monitors that the distribution of values in a column (mean, median, standard deviation) remains within expected bounds, a core technique for data drift detection.
• Volume/Row Count Rule: Validates that the number of records in a dataset is within an expected range (e.g., daily_record_count BETWEEN 1000 AND 1500), alerting to ingestion failures or surges.

These are complex, domain-specific rules that encode critical business knowledge and calculations.

• Calculated Metric Rule: Validates the correctness of a derived metric against its constituent parts (e.g., total_revenue = SUM(line_item_price * quantity)).
• Multi-Table Business Rule: Enforces logic spanning multiple datasets (e.g., the sum of all regional sales in a fact table must equal the company-wide total in a separate summary table).
• Compliance Rule: Encodes regulatory or legal requirements (e.g., data_retention_days >= 7 years for financial records). These are often implemented as business rule validation suites.

A programmatic check within a data pipeline that verifies a specific condition about the data and raises an error or warning if false. It is the fundamental building block for enforcing a Data Quality Rule.

• Implementation: Often written as a conditional statement (e.g., assert df['column'].isnull().sum() == 0).
• Scope: Can validate row counts, value uniqueness, referential integrity, or custom business logic.
• Role: Serves as the executable instantiation of a declarative rule.

A formal agreement, often codified as code, that specifies the expected schema, data types, freshness, and quality guarantees for a data product. Data Quality Rules are a core component of a data contract.

• Purpose: Establishes a clear interface between data producers and consumers.
• Content: Includes structural rules (schema), behavioral rules (freshness SLOs), and quality rules (valid value ranges).
• Enforcement: Violations of contracted rules constitute a breach, triggering alerts or blocking pipeline progression.

A collection of Data Quality Rules or assertions that define the expected properties of a specific dataset. Used in frameworks like Great Expectations.

• Composition: A suite contains multiple individual expectations (e.g., expect_column_values_to_be_unique, expect_table_row_count_to_be_between).
• Management: Suites are versioned and stored as JSON or YAML, enabling Data Quality as Code.
• Execution: Run as a batch by a Checkpoint to validate a dataset snapshot.

A Data Quality Rule whose failure prevents a data pipeline from proceeding to the next stage. It acts as a mandatory quality gate to block invalid data from propagating.

• Function: A critical assertion that halts pipeline execution or triggers a rollback.
• Examples: Verifying a primary key is unique before a merge, or ensuring a required source file has arrived.
• Outcome: Failure creates an incident that must be resolved before data flow resumes.

An approach where Data Quality Rules are specified as configuration stating what the data condition should be, rather than how to check it programmatically.

• Syntax: Uses YAML, SQL, or domain-specific languages (e.g., validates_uniqueness_of :user_id).
• Frameworks: Exemplified by dbt Tests and Soda Core checks.
• Advantage: Separates the rule definition from the execution engine, improving portability and readability.

The process of testing data against complex, domain-specific logic that defines correct business relationships. This extends Data Quality Rules beyond basic schema checks.

• Scope: Validates calculations, multi-column logic, and compliance with operational policies.
• Examples: "Total invoice amount equals the sum of line items," or "Discount cannot exceed 50% for premium customers."
• Complexity: Often requires custom SQL or Python assertions to implement.