Data Validity

Data validity is fundamentally defined by explicit rules. These are logical or syntactic constraints that data must satisfy to be considered valid. Common rule types include:

• Format Rules: Enforcing patterns like email addresses ([email protected]), phone numbers, or ISO date strings (YYYY-MM-DD).
• Range Rules: Ensuring numerical values fall within a minimum and maximum (e.g., age between 0 and 120).
• Domain Rules: Restricting values to a predefined list or set (e.g., status in ['active', 'inactive', 'pending']).
• Type Rules: Verifying data matches the expected primitive type (string, integer, float, boolean). Without these codified rules, validity cannot be objectively measured.

Validity operates on two distinct levels:

• Syntactic Validity checks if data adheres to formal structure and format. A date string '2024-13-45' is syntactically invalid because month 13 and day 45 violate the Gregorian calendar format.
• Semantic Validity (or logical validity) checks if data makes sense within a business context. The date '2024-02-30' is syntactically correct but semantically invalid, as February never has 30 days. Advanced validation systems layer semantic checks on top of syntactic ones, often using business logic or reference data to assess real-world plausibility.

Modern data engineering implements validity checks through declarative frameworks like Great Expectations, dbt tests, or Soda Core. Engineers define validation rules as code or configuration, separate from pipeline logic. For example, a great_expectations suite might declare: expect_column_values_to_be_in_set('country_code', ['US', 'CA', 'UK']). This separation allows for:

• Reusable test suites across pipelines.
• Automated documentation of data contracts.
• Centralized management of quality rules, enabling consistent enforcement at ingest, transformation, and publish stages.

Data validity is tightly coupled with schema enforcement. A schema defines the expected structure: column names, data types, and constraints (like NOT NULL). Validity checks extend beyond basic schema by enforcing richer, domain-specific rules.

• Static Schema: Traditional RDBMS enforce strict, predefined schemas at write-time, a strong form of validity control.
• Schema-on-Read: Systems like data lakes use flexible schemas applied during read, pushing validity checking to downstream consumers. This increases flexibility but risks invalid data proliferation.
• Schema Evolution: Validity rules must version alongside schemas to handle backward-compatible and breaking changes without causing pipeline failures.

Invalid data acts as a silent pipeline toxin. Its impact propagates and amplifies:

• Analytics & BI: Invalid categorical values (e.g., 'TX' and 'Texas' for the same state) fragment reports and skew aggregations.
• Machine Learning: Models trained on invalid ranges or formats learn incorrect patterns, degrading predictive accuracy. An invalid -1 age value corrupts feature distributions.
• Operational Systems: Invalid foreign keys break referential integrity, causing application errors. An invalid customer_id format can crash a billing system. The cost of remediation increases exponentially the further invalid data travels, making early validation a critical cost-control measure.

Effective validity is enforced through a defense-in-depth strategy across the data lifecycle:

1. Ingest/Point of Entry: Validate raw data against ingestion contracts using tools like Apache Schematron or custom pre-processors. Reject or quarantine invalid records immediately.
2. Transformation: Apply transformational validity checks within ETL/ELT jobs (e.g., dbt tests) to ensure cleaning and business logic produce valid outputs.
3. Publish/Consumption: Implement final consumer-facing checks before data is exposed to analytics or ML models, often via data quality gates. This staged approach contains issues, provides clear failure isolation, and ensures only valid data is promoted to trusted zones.

These rules enforce correct textual or structural patterns for data fields. They are fundamental for data parsing and interoperability between systems.

• Email Addresses: Must contain a single @ symbol and a valid domain with a period (e.g., [email protected]).
• Phone Numbers: Must match a country-specific pattern (e.g., US: (XXX) XXX-XXXX).
• Dates & Timestamps: Must conform to a specified format like ISO 8601 (YYYY-MM-DD) and represent a real calendar date.
• Social Security Numbers (US): Must follow the pattern XXX-XX-XXXX where X is a digit, and cannot be all zeros in any segment.
• URLs: Must begin with a valid protocol (http://, https://) and contain a domain name.

These rules ensure numerical or date values fall within plausible minimum and maximum limits, catching outliers and impossible values.

• Age of a Customer: Must be between 18 and 120 years.
• Product Discount Percentage: Must be between 0% and 100%.
• Order Quantity: Must be a positive integer (greater than 0).
• Transaction Amount: Must be greater than $0.01 and less than a defined maximum (e.g., $1,000,000 for fraud prevention).
• Sensor Temperature Reading: Must be within the operational range of the hardware (e.g., -40°C to 125°C).

These rules maintain the consistency of relationships between datasets by validating that foreign keys have matching primary keys in related tables.

• Order customer_id: Must exist in the customers table's id column.
• Invoice product_sku: Must exist in the products table's sku column.
• Employee department_code: Must exist in the departments lookup table.
• Shipping address_id: Must exist in the addresses table for the given user. Violation of these rules indicates broken relationships, often caused by deletion or missing reference data.

These rules constrain a field's value to a predefined list of allowed options, ensuring categorical consistency.

• Country Code: Must be a valid two-letter ISO 3166-1 alpha-2 code (e.g., US, GB, JP).
• Order Status: Must be one of: pending, processing, shipped, delivered, cancelled.
• Product Category: Must belong to the master list of categories defined in the product taxonomy.
• Payment Method: Must be one of: credit_card, debit_card, paypal, bank_transfer.
• HTTP Status Code: Must be a standard code like 200, 404, 500.

These are complex rules where the validity of one field depends on the value of another, enforcing business logic.

• If shipment_type is 'express', then delivery_date must be within 2 days of order_date.
• If payment_method is 'credit_card', then card_expiry_date must be in the future.
• If age is under 18, then guardian_consent_flag must be 'true'.
• end_date must be on or after start_date for any project or subscription record.
• discount_amount cannot be greater than subtotal_amount.

These rules prevent duplicate entries for fields or combinations of fields that must be unique within a dataset.

• Primary Key Fields: A column like user_id or transaction_id must be unique across all rows.
• Natural Keys: employee_email or product_sku must be unique within an active dataset.
• Composite Uniqueness: The combination of user_id and course_id must be unique in an enrollment table to prevent duplicate enrollments.
• Session Identifiers: A session_token must be globally unique to prevent collision. Enforcement is critical for data integrity and preventing operational errors like double-charging.

Data accuracy measures the degree to which data values correctly represent the real-world entities or events they are intended to describe. While validity checks for format, accuracy assesses truthfulness.

• Example: A customer's birth date may be a valid date format (validity), but be incorrect by 10 years (inaccuracy).
• Key Difference: Validity is syntactic; accuracy is semantic. A value can be valid but inaccurate.

Data completeness measures the proportion of expected data values that are present and non-null. It ensures that datasets contain all necessary records and fields.

• Calculated as: (Number of non-null values / Total expected values) * 100.
• Relationship to Validity: A field can be 100% complete (no nulls) but contain invalid entries (e.g., 'N/A' in a date field). Both dimensions must be monitored.

Schema validation is the technical process of verifying that data conforms to a predefined structural and type definition, often using tools like JSON Schema, Avro, or Protobuf. It is a primary mechanism for enforcing data validity.

• Enforces: Data types (integer, string), required fields, allowed value ranges, and nested object structures.
• Implementation: Typically applied at pipeline ingestion points as a data quality gate to reject malformed records.

Referential integrity is a constraint that ensures relationships between tables remain consistent. It validates that a foreign key value in one table has a matching primary key value in a related table.

• A Validity Subtype: It enforces a specific validity rule about relational consistency.
• Example: An order record with a customer_id of 'C999' is invalid if no customer with that ID exists in the customers table.

Business rule validation extends syntactic validity by checking data against domain-specific logical constraints. These rules encode organizational policies and operational logic.

• Examples: discount_percentage <= 100, ship_date >= order_date, account_status must be 'active' for a transaction.
• Implementation: Often codified in data testing frameworks (e.g., dbt tests, Great Expectations) and executed as part of pipeline orchestration.

A data quality gate is an automated checkpoint within a data pipeline that evaluates metrics like validity, completeness, or accuracy. It can halt processing, quarantine bad data, or trigger alerts if thresholds are violated.

• Purpose: Prevents invalid data from propagating to downstream consumers like dashboards or machine learning models.
• Common Tools: Implemented using open-source libraries (e.g., Soda Core, Great Expectations) or features within data observability platforms.