Data Quality Rule

A data quality rule is a declarative statement of a condition that data must meet, not an imperative procedure. It is expressed in a way that can be automatically evaluated as true or false for a given dataset. This testability is fundamental, enabling automation through validation frameworks.

• Example: "The customer_id field must be unique across all records in the customers table."
• Implementation: This rule can be executed as a SQL query (SELECT COUNT(*) FROM customers GROUP BY customer_id HAVING COUNT(*) > 1) or configured in a data quality tool.

Every rule has a defined scope—the specific data asset, column, table, or pipeline stage to which it applies. Context includes the business meaning and criticality of the rule (e.g., blocking vs. warning).

• Technical Scope: Applies to table transactions, column amount, for records where status = 'completed'.
• Business Context: This rule ensures financial reporting accuracy; a failure is a P1 incident.
• Temporal Context: A freshness rule (data must be updated within 1 hour) has an inherent time-based scope.

The execution of a data quality rule produces a quantifiable metric. This is not just a pass/fail flag but a measurement that can be tracked over time.

• Primary Metric: Rule failure rate (e.g., 0.5% of records violated the uniqueness constraint).
• Derived Metrics: Percentage of null values, number of distinct values outside an expected range.
• Trend Analysis: These metrics allow teams to monitor data drift and establish Service Level Objectives (SLOs) for data quality, such as "99.9% of records must pass all validity rules."

Effective rules are integrated into the data lifecycle, not applied ad-hoc. They are embedded within pipelines, orchestration tools, and observability platforms.

• Pipeline Integration: Executed as a step in an ETL/ELT job (e.g., using dbt tests, Great Expectations checkpoints).
• Proactive Monitoring: Continuously evaluated on streaming data or scheduled batches.
• Incident Management: Failures trigger alerts, create tickets, and can block promotion of data to production environments, forming a core part of Data Reliability Engineering.

A rule is a governed asset with clear ownership, versioning, and understood lineage. It should be documented in a data catalog or governance platform.

• Ownership: Assigned to a data steward or engineering team.
• Versioning: Changes to rule logic are tracked to understand their impact over time.
• Lineage: The rule's downstream impact is known—which reports, models, or applications depend on data validated by this rule. This connects directly to data contract enforcement.

Rules categorize into several fundamental types based on the dimension of quality they enforce:

• Validity/Format: Data conforms to a defined syntax (e.g., regex for email, UTF-8 validation, correct date format).
• Completeness: Required fields are not null (nullability check).
• Uniqueness: A column or combination of columns contains no duplicate values.
• Accuracy/Consistency: Values match a trusted source or adhere to business logic (e.g., end_date must be after start_date).
• Freshness/Timeliness: Data is updated within an expected timeframe.
• Referential Integrity: A foreign key value exists in the referenced primary key column.
• Distributional: Statistical properties (mean, standard deviation) remain within expected bounds, guarding against schema drift and outliers.

The most fundamental implementation of a data quality rule is the CHECK constraint in SQL databases. It is a rule enforced at the database engine level on a table column.

• Declarative Syntax: Defined in CREATE TABLE or ALTER TABLE statements (e.g., CHECK (age >= 0)).
• Engine Enforcement: The DBMS automatically validates the constraint on INSERT and UPDATE operations, rejecting violating transactions.
• Performance: Evaluation is highly optimized but can add overhead for complex rules.
• Limitation: Rules are scoped to a single row; they cannot reference other rows or tables (unlike triggers). This is a core tool for ensuring domain integrity.

The process of verifying that a data structure's format, data types, and organization conform to a predefined formal specification, or schema. This is a foundational check that ensures structural integrity before applying more complex business logic rules.

• Examples: Validating that a JSON object contains all required fields with correct types (string, integer, array) as defined by a JSON Schema.
• Purpose: Acts as a first line of defense against malformed data entering a pipeline.

A formal, versioned agreement between data producers and consumers that explicitly defines the schema, semantics, quality guarantees (SLOs), and evolution policies for a data product. It moves validation from an informal expectation to a machine-testable obligation.

• Key Components: Schema definition, data freshness guarantees, backward/forward compatibility rules, and deprecation policies.
• Benefit: Reduces pipeline breaks by establishing clear, enforceable interfaces, shifting validation left in the development lifecycle.

The overarching property of data being accurate, consistent, and reliable throughout its entire lifecycle. Data quality rules are the technical enforcement mechanisms that uphold data integrity.

• Enforced By: Constraints (e.g., PRIMARY KEY, FOREIGN KEY), validation rules, and consistency checks.
• Broader Than Quality: While data quality often focuses on content, integrity also encompasses structural and relational correctness, especially in database systems.

The automated analysis of an existing dataset to discover its structure, content, and relationships. Profiling generates statistics and metadata that inform the creation of precise data quality rules.

• Common Profiling Metrics: Data type detection, pattern analysis (e.g., email formats), value distributions, uniqueness, and null value percentages.
• Workflow: Profiling is typically performed first to understand the data, followed by rule definition based on the discovered patterns and anomalies.

A centralized service for storing, managing, and enforcing schemas (e.g., Avro, Protobuf, JSON Schema) in data streaming architectures like Apache Kafka. It enables schema validation at the point of data production or consumption.

• Core Function: Provides schema versioning, compatibility checks (backward/forward), and serialization/deserialization (serde) clients.
• Role in Validation: Ensures all data published to or read from a stream adheres to a registered, compatible schema, acting as a governance layer for data quality rules.

The practice of applying software testing principles to data assets, using code to programmatically validate data integrity and business logic. Data quality rules are the test assertions in this framework.

• Frameworks: Tools like dbt tests, Great Expectations, and Deequ allow engineers to define suites of validation rules (tests) that run as part of CI/CD or pipeline orchestration.
• Outcome: Shifts validation from a manual, ad-hoc process to a deterministic, automated checkpoint that can fail pipelines and alert teams.