Data Downtime

These are direct failures in the data processing infrastructure that halt or corrupt data flow. They are often the most visible cause of downtime.

• Job Failures: A scheduled data ingestion or transformation job (e.g., in Apache Airflow, dbt) crashes due to code errors, resource exhaustion, or dependency failures.
• Infrastructure Outages: Loss of a cloud data warehouse, streaming service (Kafka), or compute cluster.
• Schema Mismatches: Upstream schema changes (new column, altered data type) break downstream transformations that rely on a specific structure, causing jobs to fail.
• Resource Contention: Concurrent jobs or queries exhaust memory, CPU, or I/O bandwidth, leading to timeouts or partial data loss.

Unannounced or breaking modifications to the systems that generate the source data. These changes propagate silently, corrupting data before it even enters the primary pipeline.

• API Contract Breaks: A third-party SaaS application (e.g., Salesforce, Stripe) changes its API response format, field names, or authentication without warning.
• Database Migrations: An operational database used as a source is altered—tables are renamed, columns are dropped, or legacy data is purged.
• Human Process Changes: A business team alters how they manually enter data into a CRM or spreadsheet, introducing new formats or nullifying previously mandatory fields.
• Latent Source Bugs: A bug in a production application begins writing incorrect or malformed data to a log or database table.

The data arrives but fails predefined business logic or integrity checks. This is often the most insidious cause, as pipelines may continue running while serving 'bad' data.

• Accuracy Breaches: Key metrics (e.g., revenue, user count) deviate from known thresholds or correlated sources.
• Completeness Degradation: Critical fields experience a sudden spike in null rates, or expected daily record volumes drop significantly.
• Freshness Staleness: Data fails to update within the required SLA, making dashboards and models operate on outdated information.
• Uniqueness & Duplication: Primary key violations or an explosion of duplicate records distort aggregate analyses.

The statistical properties or real-world meaning of the data changes over time, rendering existing data products and models inaccurate, even if the data is technically 'correct'.

• Feature Distribution Shift: The underlying population changes (e.g., a new user demographic signs up), altering the distribution of model input features like age or purchase frequency.
• Label Meaning Change: The definition of a target variable evolves. For example, the business redefines an 'active user' from 'logged in once per week' to 'completed a specific action'.
• Seasonality & Trend Breaks: Expected cyclical patterns (daily, weekly) are disrupted by external events (holidays, outages), causing models to misinterpret normal operations as anomalies.
• Covariate Shift: The relationship between input features and the target variable changes, degrading model performance without any apparent data errors.

Failures in the scheduling, coordination, and dependency management of complex data workflows. A single delayed or mis-ordered task can cascade.

• Dependency Deadlocks: Job A waits for Job B, which is waiting for Job A, causing a deadlock and halting the entire DAG.
• Late Arriving Data: A critical upstream dataset is delayed, causing all downstream dependent jobs to either fail, time out, or process incomplete data.
• Incorrect Execution Order: A transformation job runs before its source data ingestion job has completed, leading to processing of a partial or previous day's data.
• Versioning Conflicts: A new version of a shared dataset or library is deployed, breaking compatibility with consumers who haven't updated their code.

Human errors in the deployment of code, changes to environment variables, or misconfigurations of services that govern data processing.

• Secret Rotation Failures: Database credentials or API keys expire or are rotated without updating the data pipeline configuration, causing authentication failures.
• Environment Mismatches: Code tested in a staging environment behaves differently in production due to unaccounted-for configuration differences (e.g., memory limits, timezone settings).
• Incremental Logic Flaws: A bug in the logic identifying 'new' data for incremental processing causes historical data to be reprocessed incorrectly or new data to be missed entirely.
• Rollback Failures: A failed deployment cannot be cleanly rolled back, leaving the pipeline in a broken or inconsistent state.

A Data SLO is a formal, quantitative target for the reliability of a data product or pipeline. It defines the acceptable level of service, typically expressed as a percentage of time that specific data quality metrics (like freshness or completeness) must meet a predefined threshold. Data downtime is the direct inverse of an SLO; it quantifies the time spent outside of the agreed-upon service level.

• Example: "The customer analytics table must have 99.9% freshness (data less than 1 hour old) over a 30-day rolling window."
• Purpose: Provides a clear, business-aligned target for data engineering teams and forms the basis for an error budget.

Mean Time To Detect (MTTD) is a critical data reliability metric that measures the average duration between the onset of a data quality incident and its initial discovery by monitoring systems or teams. A high MTTD directly increases data downtime, as the period of unusable data extends undetected.

• Key Driver: Effective data observability tooling—including anomaly detection on freshness, volume, and schema—is essential to minimize MTTD.
• Impact: Reducing MTTD is often the first and most significant lever for decreasing total data downtime, as the clock stops only when the issue is identified.

Mean Time To Resolve (MTTR) measures the average duration from the detection of a data quality incident to its full resolution and the restoration of normal service. Alongside MTTD, MTTR is a primary component of total data downtime.

• Factors Influencing MTTR: Complexity of root cause (e.g., schema drift vs. source system outage), quality of data lineage mapping for impact analysis, and efficiency of data incident management processes.
• Goal: Engineering practices like automated data validation gates, robust pipeline design, and clear runbooks aim to reduce MTTR.

A Data Error Budget is the explicit, allowable amount of time that a data product can fail to meet its Data SLOs before triggering a formal review or halting feature development. It is calculated as: (100% - SLO%) * Time Period. Consumed error budget equals data downtime.

• Operational Use: Teams can move quickly and accept some risk as long as error budget remains. Exhausting the budget mandates a focus on reliability work over new features.
• Example: For a 99.9% monthly SLO, the error budget is 0.1% of the month, or approximately 43.2 minutes of allowed downtime.

Data Freshness is a core data quality dimension that measures the age of data at the point of consumption, calculated as the time elapsed since the source data was last updated. Stale data is a primary cause of data downtime, as outdated information is unfit for use.

• Measurement: Often tracked as a Data SLI (Service Level Indicator), such as "95th percentile data latency < 5 minutes."
• Monitoring: Data freshness and latency monitoring systems alert when data update cycles are delayed, directly flagging potential downtime events.

A Data Health Index is a synthesized, often business-facing score that aggregates multiple technical data quality metrics (like accuracy, completeness, freshness) and pipeline statuses to indicate the overall fitness-for-use of a data asset. A declining health index is a leading indicator of impending data downtime.

• Composition: May combine automated data quality scores (DQS), coverage metrics for checks, and breach of data quality gates.
• Purpose: Provides a single, at-a-glance view of data asset reliability for stakeholders beyond the data engineering team.