Data Completeness

The null rate is the foundational metric for data completeness, calculated as the percentage of null or missing values within a specific column, table, or dataset. It provides a direct, quantitative measure of missingness.

• Calculation: (Number of Null Values / Total Number of Rows) * 100.
• Example: A customer email column with 950 non-null values out of 1000 total rows has a null rate of 5%.
• Use Case: Used to set automated alerts; a sudden spike in a column's null rate often indicates a broken data ingestion process.

Completeness is evaluated at two primary levels: record completeness and attribute completeness.

• Record Completeness: Measures if an entire expected data record is present. For example, a daily sales table should have one record per store; missing a store's record for a day indicates a record-level completeness failure.
• Attribute Completeness: Measures if specific fields (attributes) within a record are populated. A customer record might be present, but if the customer_id field is null, it suffers from attribute-level incompleteness.
• Critical Distinction: A dataset can have 100% record completeness but poor attribute completeness, making this distinction vital for root cause analysis.

Completeness requirements are defined by business rules classifying fields as mandatory or optional.

• Mandatory Fields: Core attributes essential for data utility and integrity, such as primary keys (user_id), transaction timestamps, or monetary amounts. A null in a mandatory field typically renders a record unusable.
• Optional Fields: Supplementary attributes where missing values are acceptable for certain analyses, such as a customer's middle name or secondary phone number.
• Governance: Data contracts and schema definitions explicitly tag fields to automate completeness checks. Violations in mandatory fields trigger high-severity alerts.

Data completeness is not static; it must be monitored as a time-series metric to detect degradation. This involves tracking completeness trends and volatility.

• Trend Analysis: A gradually increasing null rate in a key column may indicate a deprecated data source or a changing extraction logic.
• Volatility Detection: Sudden, sharp drops in completeness (e.g., from 99% to 70%) are critical incidents, often caused by pipeline failures or API changes.
• Baseline Comparison: Current completeness metrics are compared against a data quality baseline—a historical profile of expected null rates—to distinguish normal variation from significant drift.

Some data incompleteness is not revealed by null values alone but through logical inference, known as implied missingness.

• Example: A database of support tickets should have a resolved_at timestamp if the status field is 'closed'. A 'closed' ticket without this timestamp is incomplete, even though no field is technically null.
• Derived Metrics: Completeness checks must therefore evaluate business logic and cross-field dependencies, often implemented as data quality rules within validation frameworks.
• Referential Integrity: A related concept where a foreign key value must have a matching primary key in a parent table; its absence implies missing relational data.

Incomplete data propagates errors, causing failures in dependent processes, a phenomenon known as data downtime.

• Analytics & BI: Aggregations (like SUM, COUNT) on incomplete columns produce misleading metrics. A missing revenue value is excluded from sums, underreporting totals.
• Machine Learning: Models trained on datasets with systematic missingness learn biased patterns, degrading predictive accuracy. Many algorithms cannot handle null values without explicit imputation.
• Operational Systems: Automated decisions (e.g., fraud scoring, customer personalization) fail or produce erroneous outputs when key input fields are empty.
• Mitigation: Implementing data quality gates that block incomplete data from progressing prevents corruption of downstream assets.

Measures the degree to which data values correctly represent the real-world entities or events they describe. It is distinct from completeness, as a complete dataset can still contain inaccurate values.

• Key Focus: Correctness and truthfulness of the data.
• Example: A customer record with a complete address (completeness) but an incorrect postal code (inaccuracy).
• Measurement: Often validated against a trusted source of truth or through business logic rules.

Measures the degree to which data values conform to a defined set of syntactic rules, formats, or allowable value ranges (a domain). It is a prerequisite for both accuracy and completeness.

• Key Focus: Adherence to structural and format rules.
• Example: An email field containing a value like user@company is invalid (missing .com), even if the value is present (complete).
• Common Checks: Data type, regex patterns, value whitelists/blacklists, and range constraints.

A specific, operational metric used to quantify data completeness. It calculates the percentage of null or missing values within a specific column, table, or dataset.

• Calculation: (Number of Null Values / Total Number of Rows) * 100.
• Usage: Directly feeds into completeness assessments and SLOs. A high null rate in a critical field indicates a severe completeness issue.
• Monitoring: Often tracked with control charts to detect unexpected increases in missing data over time.

Measures the logical coherence and absence of contradictions for data values across different datasets, tables, or systems. Inconsistent data can undermine trust even in complete datasets.

• Key Focus: Logical agreement across sources and over time.
• Example: A customer's lifetime value in the CRM is $10,000, but the data warehouse reports $8,500 for the same customer and period.
• Types: Includes transactional consistency, semantic consistency, and temporal consistency.

Measures the absence of duplicate records or entities within a defined dataset or across a specified set of key fields. It ensures each real-world entity is represented once.

• Key Focus: Elimination of redundant records.
• Relation to Completeness: Duplicate records can artificially inflate counts, masking true completeness issues for unique entities.
• Measurement: Often assessed via duplicate count metrics using exact or fuzzy matching on defined keys.

A fundamental category or aspect—such as accuracy, completeness, timeliness, and validity—used to characterize, measure, and manage the quality of data. Completeness is one of the six core dimensions defined by most frameworks (e.g., DAMA-DMBOK).

• Purpose: Provides a structured vocabulary for discussing and assessing data quality.
• Framework: Dimensions are often measured by specific metrics (like null rate for completeness) and managed via data quality rules.
• Holistic View: Effective data quality programs measure and monitor across multiple dimensions simultaneously.