Data Quality Metrics

Accuracy measures the degree to which data correctly reflects the real-world entity or event it represents. It is a measure of correctness.

• Example: A customer's date of birth in a CRM system matching their official ID.
• Challenge: Often requires an external, authoritative source of truth for verification.
• Metric: Often expressed as an error rate (e.g., 99.5% of records match the verified source).

Completeness assesses the extent to which expected data is present and non-null in a dataset. It answers: 'Do we have all the data we need?'

• Measured at the record, column, or dataset level.
• Example: A required 'postal_code' field is missing for 2% of customer records.
• Impact: Missing features can cause models to fail or produce biased inferences. High completeness is critical for training robust models.

Consistency evaluates whether data is uniform and conflict-free across different datasets, tables, or within a single record. It ensures logical coherence.

• Intra-record: A patient's 'admission_date' must be before their 'discharge_date'.
• Cross-system: A customer's lifetime value in the data warehouse should match the aggregated value in the CRM.
• Format Consistency: All phone numbers follow the same national/international format (e.g., +1-xxx-xxx-xxxx).

Timeliness measures how current and up-to-date the data is relative to the task it supports. It reflects the latency between a real-world event and its availability in the dataset.

• Critical for real-time applications: Fraud detection, dynamic pricing, and sensor-based systems require data freshness measured in milliseconds or seconds.
• For batch analytics, timeliness might be measured in hours or days.
• Metric: Data Latency = (Time data is available for use) - (Time event occurred).

Uniqueness identifies the absence of duplicate records within a dataset. It ensures each real-world entity is represented only once.

• Primary cause of data inflation and skewed analytics.
• Example: A single customer with three slightly different email addresses appears as three distinct customers.
• Process: Data deduplication uses fuzzy matching on key identifiers (name, email, address) to find and merge duplicates.

Validity checks if data conforms to a defined syntax, format, type, range, or set of business rules (its schema). It is a measure of formal correctness.

• Syntax: An email address must contain an '@' symbol.
• Range: A product's 'discount_percentage' must be between 0 and 100.
• Type: A 'transaction_amount' field must be a numeric, not a string.
• Enforced via: Data validation rules during ingestion and transformation.

Data validation is the programmatic process of checking a dataset against predefined rules, schemas, and constraints to ensure its correctness, completeness, and consistency before use in training or inference. It is a proactive quality gate.

• Schema Enforcement: Validates data types, value ranges, and required fields.
• Referential Integrity: Checks consistency across related data tables.
• Custom Rule Checks: Applies business logic (e.g., 'order_date' must be before 'ship_date').
• Tools: Frameworks like Great Expectations, Pandera, or Deequ automate this process, generating validation reports.

These metrics track changes over time that degrade model performance. Data Drift occurs when the statistical properties of the input feature distribution change (e.g., average transaction value shifts). Concept Drift occurs when the relationship between inputs and the target variable changes (e.g., the definition of 'fraudulent transaction' evolves).

• Detection Methods: Statistical tests (KS-test, PSI), model-based detectors, and monitoring embedding distributions.
• Impact: Unaddressed drift leads to silent model performance decay. Monitoring these is critical for MLOps.

IAA is a statistical measure of consistency among multiple human labelers annotating the same data. It quantifies label reliability and annotation guideline clarity.

• Common Metrics: Cohen's Kappa (binary), Fleiss' Kappa (multi-annotator), Krippendorff's Alpha (robust to missing data).
• Benchmark: Kappa > 0.8 indicates excellent agreement; < 0.6 suggests guidelines need revision.
• Purpose: High IAA is a prerequisite for creating reliable ground truth datasets.

Data provenance is the complete audit trail documenting a dataset's origin, transformations, ownership, and processing steps. It is foundational for trust, reproducibility, and regulatory compliance.

• Tracks: Source systems, transformation code versions, responsible engineers, and timestamps.
• Enables: Debugging pipeline errors, reproducing model results, and fulfilling GDPR 'right to explanation' requests.
• Tools: Implemented via data lineage features in platforms like MLflow, DVC, or Apache Atlas.

Bias auditing is the systematic evaluation of a dataset or model for unfair, skewed, or discriminatory representations across demographic or contextual groups. It is a core component of algorithmic fairness.

• Dataset Metrics: Measure representation parity (e.g., gender balance), label distribution across groups, and stereotypical associations in text or images.
• Model Metrics: Evaluate disparities in performance metrics like accuracy, F1-score, or false positive rates across subgroups.
• Frameworks: Tools like Fairlearn, Aequitas, and the Hugging Face evaluate library provide standardized audits.

Data integrity refers to the accuracy, consistency, and trustworthiness of data over its entire lifecycle. It ensures data is unaltered from its source and maintains referential and entity integrity.

• Components: Accuracy (correct values), Consistency (uniform format across systems), Reliability (dependable source).
• Threats: Corruption during transfer, unauthorized alterations, software bugs.
• Enforcement: Achieved through data validation, encryption, checksums, and robust data governance policies.