Data Quality Incident

Every incident originates from a breach of one or more predefined data quality dimensions. These are the measurable attributes of data health. Common violations include:

• Freshness: Data is stale or delayed beyond the acceptable Service Level Objective (SLO).
• Completeness: Critical fields contain an unexpected volume of null or missing values.
• Validity: Data fails to conform to expected formats, types, or business rules (e.g., an email field contains a phone number).
• Uniqueness: Duplicate records appear where they should be unique, skewing aggregate counts.
• Accuracy: Data values do not correctly reflect the real-world entities they represent.
• Consistency: The same data element differs across systems or reports.

An incident is characterized by its tangible, negative effect on business operations or decision-making. This impact is what elevates a data anomaly to an incident status. Impact is typically assessed across several axes:

• Operational Disruption: Downstream ETL/ELT pipelines, dashboards, or machine learning models fail or produce erroneous outputs.
• Financial Cost: Incorrect data leads to flawed business decisions, regulatory fines, or loss of customer trust. For example, a pricing data corruption incident could directly affect revenue.
• Analytical Integrity: Reports and key performance indicators become unreliable, misleading leadership.
• Consumer Trust: Customer-facing applications display incorrect information, damaging the user experience. The Incident Severity Matrix is used to objectively classify this impact to determine response priority.

Unlike transient noise, a confirmed data quality incident triggers a formal incident management process. This structured response is a defining characteristic and includes:

• Incident Triage: Initial validation, severity classification, and assignment of ownership.
• Impact Assessment: Determining the scope of affected datasets, pipelines, and business units.
• Remediation: Executing fixes, which may involve automated rollback, pipeline restarts, or data correction scripts.
• Root Cause Analysis (RCA): A systematic investigation to identify the underlying failure in the pipeline, code, or process, not just the symptom.
• Post-Incident Review: Documenting lessons learned and creating follow-up actions to prevent recurrence, ideally conducted as a blameless postmortem.

A core characteristic is the ability to trace the incident back to a specific origin point within the data lineage. The root cause is rarely the data itself but a failure in the process that creates or moves it. Common sources include:

• Source System Changes: An upstream API schema change without proper notification.
• Pipeline Breakage: A job failure due to resource exhaustion, code bugs, or infrastructure issues.
• Schema Drift: New, unexpected columns appear, or data types change, breaking downstream transformations.
• Human Error: A flawed manual data load or incorrect configuration deployment.
• Third-Party Data Feed Degradation: A vendor's data feed becomes incomplete or malformed. Effective data observability platforms provide the lineage and dependency mapping needed for this traceability.

The lifecycle and severity of an incident are quantified using standard operational metrics. These metrics are critical for governance and continuous improvement:

• Detection Time: The delay between when the data quality breach occurred and when it was alerted on. Minimizing this is a goal of proactive monitoring.
• Mean Time to Acknowledge (MTTA): The average time for an on-call engineer to begin investigating the alert.
• Mean Time to Resolve (MTTR): The key metric measuring the average total time to fully restore data quality and pipeline operation.
• Recovery Point Objective (RPO): The maximum acceptable data loss (e.g., 'we can afford to lose up to 1 hour of data').
• Recovery Time Objective (RTO): The maximum acceptable downtime for the data service or pipeline.

It is crucial to distinguish a data quality incident from a general infrastructure or availability incident. While related, they have different primary failures:

• Data Quality Incident: The system is operational (pipeline jobs run, databases are online), but the data flowing through it is corrupt, stale, or invalid. The business logic is broken.
• System/Platform Incident: The infrastructure is down (compute cluster fails, network partition). Data cannot flow at all, but its intrinsic quality may be unaffected. A pipeline breakage often sits at the intersection: a system failure (job crash) causes a data quality incident (freshness violation). Effective Data Reliability Engineering addresses both infrastructure and data quality Service Level Objectives (SLOs) under a unified error budget.

The systematic end-to-end process for handling disruptions to data pipelines, quality, or availability. It encompasses the entire lifecycle from detection and triage through investigation, remediation, and post-incident review. Effective management minimizes business impact and drives continuous improvement in data reliability.

A structured method for identifying the fundamental, underlying reason for an incident, rather than just addressing its symptoms. In data contexts, RCA often involves tracing failures through data lineage, analyzing pipeline execution logs, and examining changes in source systems or data schemas. The goal is to implement corrective actions that prevent recurrence.

A target level of reliability or performance for a data product or pipeline, expressed as a percentage over a compliance period. For data quality, common SLOs target dimensions like:

• Freshness: "99% of dashboard data is less than 1 hour old."
• Completeness: "99.9% of customer records contain all required fields." SLOs provide the quantitative benchmark against which incidents are measured.

The allowable amount of unreliability a data engineering team can consume, defined as the gap between 100% and an SLO. If an SLO is 99.9% freshness per month, the error budget is 0.1% (or ~43 minutes of stale data). Incidents consume this budget. Exhausting the budget triggers a focus on stability over new feature development, aligning engineering priorities with reliability goals.

A structured meeting held after an incident is resolved to analyze the event. A blameless postmortem focuses on understanding systemic failures and process gaps without assigning individual fault. The output is a document detailing:

• Timeline of events.
• Root cause(s).
• Impact assessment.
• Actionable follow-up items to improve resilience. This practice is critical for fostering psychological safety and continuous learning.

A predefined set of step-by-step procedures and checklists for responding to specific, known types of data incidents. A playbook for a schema drift incident might include steps to:

1. Isolate the affected data partition.
2. Validate the new schema against downstream consumers.
3. Execute a predefined schema migration or rollback script. Playbooks standardize response, reduce mean time to resolution (MTTR), and ensure critical steps are not missed during high-pressure situations.