Data Health Score

Measures how current the data is relative to its source update cycle and business requirements. This is critical for time-sensitive decisions.

• Key Metrics: Data latency (source-to-consumer delay), update frequency, staleness thresholds.
• Example: A customer churn prediction model requires hourly updates; data older than 2 hours receives a low freshness score.
• Impact: Stale data leads to decisions based on outdated information, directly harming predictive accuracy and operational efficiency.

Assesses whether expected data records and fields are present and non-null. Missing data skews analysis and breaks downstream processes.

• Key Metrics: Percentage of null/missing values, record count vs. expected volume, field population rates.
• Example: A daily sales feed expecting 10,000 transactions flags an anomaly if only 2,000 records arrive.
• Technical Implementation: Often involves profiling jobs that compare actual distributions against declared schemas and historical baselines.

Evaluates if data correctly represents the real-world entity or event it describes and conforms to defined format and business rules.

• Key Metrics: Rule violation counts (e.g., invalid email formats), statistical drift from known benchmarks, error rates from validation suites.
• Example: A product SKU field containing alphanumeric codes receives a low score if numeric-only entries are detected.
• Enforcement: Managed by Data Quality Rule Engines that execute declarative validation logic.

Ensures data is uniform across systems and that duplicate records do not exist where they are prohibited. Inconsistencies cause reporting conflicts.

• Key Metrics: Referential integrity failures, primary key uniqueness violations, cross-system value mismatches.
• Example: A customer ID present in the orders table but missing from the master customer table indicates a broken foreign key relationship.
• Architectural Link: This component heavily relies on accurate Data Lineage Graphs to track dependencies across systems.

Tracks the origin, transformations, and movement of data. A clear lineage is essential for trust, debugging, and impact analysis.

• Key Metrics: Depth of lineage tracking, percentage of pipeline steps instrumented, broken dependency detection time.
• Function: Answers critical questions: Where did this data come from? What transformations were applied? Who consumed it?
• Observability Integration: This is a core output of a Data Observability Platform, providing the map for Automated Root Cause Analysis (RCA).

Monitors for unexpected changes in data structure (schema drift) and statistical properties (data drift) that can break models and applications.

• Key Metrics: Schema change frequency, statistical distance metrics (e.g., KL divergence, PSI) for key columns.
• ML Impact: Data Drift Detection is vital for machine learning models, as changing input distributions degrade model performance silently.
• Process: Relies on Dynamic Baseline Calculation to establish normal ranges and Statistical Anomaly Detection to flag deviations.

These are the fundamental, atomic measurements that feed into a composite Data Health Score. They provide the quantitative basis for assessing specific dimensions of data fitness.

• Core Dimensions: Freshness (timeliness), Completeness (null rate), Validity (schema adherence), Uniqueness (duplicate rate), and Accuracy (conformity to truth).
• Example: A customer table might have a Completeness score of 99.8% (based on non-null email fields) and a Freshness score of 95% (based on records updated within the last 24 hours).
• Implementation: Often calculated via SQL queries, data profiling tools, or rule engines, these metrics are the raw inputs for higher-level scoring.

A Data SLO is the target reliability for a specific data quality metric. It defines the acceptable threshold that the Data Health Score's underlying components must meet.

• Purpose: Translates business requirements into measurable, operational targets. For example, "98% of daily sales records must be complete and available by 6 AM UTC."
• Relationship to Health Score: A Health Score can be seen as an aggregate measure of SLO compliance. If multiple underlying metrics breach their SLOs, the overall Health Score will degrade.
• Error Budgets: Derived from the SLO, this is the permissible amount of unreliability (e.g., 2% incomplete records per month) before remediation is mandated.

A Data Lineage Graph is a visual and queryable map of data's journey from source to consumption. It is critical for contextualizing a low Data Health Score and performing impact analysis.

• Function: Tracks origins, transformations, movements, and dependencies. When a Health Score drops, the lineage graph identifies all upstream sources and downstream consumers affected.
• Example: A sudden drop in a derived dashboard metric's Health Score can be traced back through the lineage graph to a specific ETL job failure or a schema change in a source database.
• Technology: Implemented via metadata collection, parsing of SQL/Scripts, or instrumentation frameworks like OpenLineage.

Automated RCA uses correlation algorithms and dependency graphs to identify the probable source of a data quality incident signaled by a declining Health Score.

• Mechanism: By analyzing timestamps, error logs, and the Data Lineage Graph, the system correlates Health Score degradation with specific pipeline failures, schema drifts, or source system outages.
• Example: When the Health Score for a key table plummets, automated RCA might pinpoint a specific failed Airflow DAG task or a spike in null values from a particular API endpoint as the root cause.
• Benefit: Drastically reduces Mean Time To Resolution (MTTR) by moving engineers directly to the faulty component.

This is the execution runtime for the declarative checks that generate the quality metrics feeding the Health Score. It applies validation logic at scale.

• Function: Executes rules defined in SQL, YAML, or a domain-specific language to check for uniqueness, referential integrity, allowable value ranges, and custom business logic.
• Output: Each rule execution produces a pass/fail result and often a quantitative measure (e.g., 5 failed records), which becomes an input for metric and Health Score calculation.
• Examples: Tools like Great Expectations, Soda Core, or custom dbt tests often function as rule engines within a broader observability platform.

DRE is the engineering discipline that operationalizes concepts like the Data Health Score, SLOs, and error budgets to systematically improve data system trustworthiness.

• Principle: Applies Site Reliability Engineering (SRE) practices to data infrastructure. The Data Health Score is a key DRE dashboard metric.
• Practices: Includes defining SLOs, calculating error budgets, implementing automated remediation, and conducting blameless post-mortems for data incidents.
• Goal: To shift from reactive firefighting to proactive, quantified management of data pipeline reliability, using the Health Score as a north-star metric.