Data Quality Metric

Accuracy measures the degree to which data correctly represents the real-world entity or event it is intended to model. It is the closeness of a data value to its true, verifiable value.

• Key Metrics: Error rate (percentage of incorrect values), distance from a known source of truth.
• Example: A customer's date of birth in a CRM system matching their government-issued ID.
• Challenges: Often requires an external, authoritative source for verification, making it one of the most difficult dimensions to measure at scale.

Completeness assesses the extent to which expected data values are present and not missing (null) in a dataset. It verifies that all required records and fields are populated.

• Key Metrics: Percentage of non-null values for a field, count of missing mandatory records.
• Example: An e-commerce order record must have values for order_id, customer_id, product_sku, and order_total.
• Rule Types: Checks for nullability as defined by the schema and validates against business rules for mandatory fields.

Consistency evaluates the uniformity of data across different systems, tables, or reports. It ensures that data does not contradict itself when compared from different sources or points in time.

• Key Metrics: Number of rule violations across systems, referential integrity error count.
• Example: The total revenue reported in a sales database matching the aggregated revenue in the finance system for the same period.
• Forms: Includes logical consistency (business rules) and format consistency (adherence to a standard pattern like date format).

Timeliness measures how current and up-to-date data is for its intended use. It combines concepts of data freshness (age of data) and latency (delay in availability).

• Key Metrics: Data freshness (time since last update), pipeline latency (source-to-consumer delay).
• Example: Stock trading algorithms require price data with sub-millisecond latency, while a monthly financial report may tolerate a 24-hour delay.
• SLOs: Often governed by Service Level Objectives (SLOs) defining acceptable freshness windows for data products.

Validity confirms that data conforms to the defined syntax, format, type, and range specified by its schema or business rules. It checks if data is well-formed and within allowable parameters.

• Key Metrics: Percentage of records passing schema validation, count of format errors.
• Example: An email field must match a standard email regex pattern; a temperature reading must be a number between -50 and 200.
• Mechanisms: Enforced through schema validation, regex validation, and range checks.

Uniqueness verifies that each data entity (record) is represented only once within a dataset or system, with no unintended duplicates. It is critical for master data and key-based operations.

• Key Metrics: Duplicate record count, percentage of unique values for a key field.
• Example: Ensuring a customer_id or social_security_number appears only once in a customer master table.
• Techniques: Enforced via database constraints (UNIQUE, PRIMARY KEY) and monitored through duplicate detection algorithms using fuzzy matching for non-exact duplicates.

A data quality rule is a formal, testable assertion that defines a specific constraint or condition data must satisfy to be considered fit for use. It operationalizes a quality dimension into executable logic.

• Examples include: format checks (e.g., email regex), range validations (e.g., age > 0), referential integrity checks, and uniqueness constraints.
• Rules are the implementation mechanism for data quality metrics; a metric like "completeness" is calculated by evaluating the percentage of records that pass a "not null" rule.
• They are foundational to automated data testing and are often codified in data contracts.

A data contract is a formal agreement between data producers and consumers that specifies the schema, semantics, quality guarantees, and service-level expectations for a data product. It makes quality commitments explicit.

• Key components: the exact schema (structure, types), data quality rules and associated metrics (e.g., freshness SLO < 1 hour), and semantics.
• It shifts quality validation left, enabling producers to test against the contract before data is published.
• Acts as the single source of truth, preventing schema drift and breaking changes that degrade downstream data integrity.

Schema validation is the process of verifying that a data structure's format, data types, and structural constraints conform to a predefined formal specification, or schema. It is a primary mechanism for enforcing structural data quality.

• Schemas can be defined using standards like JSON Schema, Avro Schema, Protocol Buffers, or XML Schema (XSD).
• Validation checks ensure fields exist, have the correct type (e.g., integer, string), and adhere to defined nullability constraints.
• A centralized schema registry often manages these schemas and performs validation in streaming pipelines to prevent malformed data.

Data profiling is the automated, statistical analysis of an existing dataset to discover its true structure, content, and quality characteristics. It is the investigative process that informs the definition of data quality metrics and rules.

• Generates metadata including: data type inference, value frequency distributions, pattern recognition, null value counts, and basic statistical summaries (min, max, mean).
• Profiling results directly feed into defining baseline metrics for completeness, uniqueness, and validity.
• It is a critical first step in data discovery and is a core function of data observability platforms and data catalogs.

Automated data testing is the practice of applying software testing principles to data pipelines, using code to programmatically validate data integrity, business logic, and quality rules. It provides continuous verification of data quality metrics.

• Test types: Unit tests for individual data transformations, integration tests for pipeline stages, and quality tests that assert data quality rules (e.g., assert df['id'].is_unique()).
• Frameworks like Great Expectations, dbt, and Deequ codify tests that run as part of CI/CD or pipeline orchestration.
• Essential for Data Reliability Engineering, enabling error budgets and SLOs for data products.

Data integrity is the overarching property of data being accurate, consistent, and reliable throughout its entire lifecycle. It is the ultimate goal enforced by data quality metrics, validation, and governance processes.

• Encompasses:
  • Physical integrity: Protection from corruption during storage and transfer.
  • Logical integrity: Enforcement of structural rules via schema validation and database constraints (e.g., PRIMARY KEY, FOREIGN KEY for referential integrity).
• Data quality metrics (accuracy, consistency) are direct measures of logical integrity.
• Maintained through data cleansing, normalization, and ETL validation processes.