Data Quality Dimension

Accuracy measures the degree to which data values correctly represent the real-world entities or events they are intended to describe. It is the closeness of a data value to a verifiably true source or an accepted standard of correctness.

• Key Challenge: Often requires an external, authoritative source of truth for validation.
• Example: A customer's date of birth in a CRM system matching their government-issued ID.
• Measurement: Typically involves direct comparison to a trusted reference dataset or real-world verification.

Completeness measures the proportion of expected data values that are present and non-null within a dataset, table, or specific field. It assesses whether all required data is available for use.

• Scope: Can be measured at the record level (mandatory fields), column level, or dataset level.
• Example: A product catalog where 95% of records have a value for the 'manufacturer' field.
• Critical for: Analytics and machine learning, where missing values can bias results or cause model failures.

Consistency measures the logical coherence and absence of contradictions for data values across different datasets, tables, or systems. It ensures data adheres to defined business rules and maintains uniformity across representations.

• Types: Includes logical consistency (e.g., end_date must be after start_date) and cross-system consistency (e.g., customer status is the same in CRM and billing systems).
• Example: The total sales figure in a daily summary report matching the sum of all individual transaction records for that day.
• Enforced by: Referential integrity constraints and cross-pipeline validation rules.

Validity measures the degree to which data values conform to a defined set of syntactic rules, formats, or allowable value ranges (a domain). It checks if data is in the correct form and structure for its intended use.

• Focus: Syntax and format, not necessarily real-world correctness (which is accuracy).
• Examples: Email addresses must contain an '@' symbol; country_code must be a valid two-letter ISO 3166-1 alpha-2 code; age must be a positive integer.
• Validated by: Schema definitions, regular expressions, and lookup tables against allowed value lists.

Uniqueness measures the absence of duplicate records or entities within a defined dataset or across a specified set of key fields. It ensures each real-world entity is represented only once for a given context.

• Measurement: Often expressed as a duplicate count or a uniqueness ratio (e.g., 99.8% of customer records are unique based on customer_id).
• Challenge: Requires defining a business key or composite key that uniquely identifies an entity.
• Impact: Duplicates can lead to overcounting in analytics, erroneous customer communications, and operational inefficiencies.

Timeliness measures the availability of data within a required timeframe relative to the event it describes. Freshness (a related aspect) measures the age of data at the point of consumption.

• Timeliness Example: Stock trade data must be available to the risk system within 100 milliseconds of execution.
• Freshness Example: A dashboard showing 'last updated 5 minutes ago' for hourly sales data.
• Key Metric: Data Latency—the delay between a data event and its availability in a target system. Critical for real-time decisioning and operational reporting.

A Data Quality Score (DQS) is a composite, often weighted, metric that aggregates measurements from multiple individual quality dimensions into a single numerical indicator. It provides a holistic view of dataset health.

• Calculation: Typically involves normalizing scores from dimensions like accuracy, completeness, and timeliness, then applying business-defined weights.
• Purpose: Enables at-a-glance assessment, trend tracking over time, and prioritization of remediation efforts.
• Example: A customer table might have a DQS of 87/100, derived from 95% completeness, 90% email validity, and 99% referential integrity.

A Data Service Level Objective (Data SLO) is a target level of reliability explicitly defined for a data product or pipeline. It represents a formal agreement on the required quality standard.

• Structure: Defined as a percentage of time specific data quality metrics must meet a threshold (e.g., "Data freshness < 1 hour for 99.9% of daily runs").
• Function: Shifts quality management from ad-hoc checks to engineered, measurable reliability. It is paired with a Data Service Level Indicator (SLI) for measurement.
• Accountability: Forms the basis for Data Error Budgets, defining allowable downtime before triggering incidents.

A Data Quality Gate is an automated checkpoint embedded within a data pipeline that programmatically evaluates one or more quality metrics before allowing processing to proceed.

• Mechanism: Executes validation rules, statistical checks, or threshold comparisons on the data in flight.
• Actions: Can halt the pipeline, trigger alerts, or route data to a quarantine zone if violations are detected.
• Implementation: Critical for preventing corrupt or low-quality data from propagating to downstream consumers and models, enforcing quality-as-code principles.

A Data Quality Baseline is a recorded set of metric values that establishes the expected, normal state of a dataset. It serves as the reference point for all future monitoring and anomaly detection.

• Components: Includes statistical properties (distributions, mean, variance), rule pass rates, and volume counts captured during a known-good period.
• Critical Use: Enables the detection of data drift and schema drift by comparing current pipeline outputs against this historical baseline.
• Management: Baselines must be updated periodically to reflect legitimate business evolution, avoiding false positives.

Statistical Process Control (SPC) for Data is a methodology that applies industrial quality control techniques to monitor data generation processes. It uses statistical tools to distinguish normal variation from significant anomalies.

• Primary Tool: The Control Chart, which plots a quality metric (e.g., null rate) over time with calculated control limits (upper control limit and lower control limit).
• Analysis: Points outside control limits or showing non-random patterns indicate a special-cause variation, signaling a process issue requiring investigation.
• Advanced Metric: The Process Capability Index (Cpk) quantifies how well a stable data process can output values within specified tolerance limits.

Data Downtime is a business-impact metric that quantifies the total period a dataset or data product is inaccurate, missing, stale, or otherwise unfit for its intended use.

• Measurement: Can be expressed in absolute time (e.g., 120 minutes of downtime this month) or as a percentage of total availability.
• Relation to SLOs: Directly linked to the consumption of a Data Error Budget. Excessive downtime indicates SLO violations.
• Components: Aggregates the impact of incidents measured by Mean Time To Detect (MTTD) and Mean Time To Resolve (MTTR).