Data Completeness SLO

The Completeness SLI is the specific, measured metric used to evaluate compliance with the SLO. It is typically expressed as a percentage, calculated as (Count of Complete Records / Total Expected Records) * 100. A complete record is defined as one where all mandatory fields are populated with non-null values. For example, an e-commerce order pipeline might define completeness as the percentage of daily orders where order_id, customer_id, product_sku, and order_amount are all present.

This component explicitly defines what is being measured. It specifies the exact data product (e.g., 'daily_customer_transactions table'), data source, and scope of the measurement. Key decisions include:

• Record-level vs. Field-level: Is completeness measured for entire records or for specific critical fields?
• Temporal Scope: Is it measured per batch (hourly/daily), per streaming window, or continuously?
• Boundary Conditions: How are late-arriving data or schema evolution handled? Clear scope prevents ambiguity in SLI calculation.

This defines the numerical target (the 'O' in SLO) and the time period over which compliance is evaluated. A common formulation is: "99.5% of records in the daily batch must be complete, measured over a 30-day rolling window."

• Target Threshold: The minimum acceptable SLI value (e.g., 99.9%, 95%).
• Compliance Window: The period for aggregation (e.g., 28 days, 1 quarter). A rolling window is preferred over a calendar period to provide a consistent, up-to-date view of reliability.

The Error Budget is derived directly from the SLO target. If the SLO is 99.5% completeness, the error budget is the remaining 0.5% of incomplete data allowed over the compliance window. An Error Budget Policy dictates how this budget is managed:

• Consumption Rate: How fast is the budget being used? A high burn rate signals a serious issue.
• Governance Actions: What happens when the budget is depleted? Policies may trigger a freeze on schema changes, mandate root cause analysis, or prioritize remediation work over new feature development for the data pipeline.

This is the operational heart of the SLO. It codifies the business rules for what constitutes 'complete' versus 'null' or missing data.

• Explicit Nulls: Are empty strings, zeros, or placeholder values like 'NULL' considered valid data or treated as missing?
• Conditional Completeness: Are some fields only mandatory based on the value of other fields?
• Validation Rules: This logic is implemented in data quality checks, often using SQL COUNT and CASE statements or frameworks like Great Expectations or Soda Core to compute the SLI automatically.

A Data Completeness SLO must be integrated into the operational monitoring and alerting stack. This involves:

• Automated SLI Calculation: Continuously computing the completeness metric from pipeline metadata or by running validation queries.
• Dashboard Visualization: Displaying current SLI value, trend over time, and remaining error budget.
• Proactive Alerting: Setting alerts based on burn rate thresholds (e.g., "Alert if error budget will be exhausted in 24 hours") rather than just point-in-time breaches, enabling teams to respond before the SLO is violated.

A Service Level Objective (SLO) is a quantitative, internal target that defines the acceptable level of reliability for a specific service metric over a defined period. For data, this translates to targets for dimensions like freshness, completeness, and correctness. An SLO is the cornerstone of a data reliability program, providing a clear, measurable goal against which performance is judged.

A Service Level Indicator (SLI) is the quantitative measure used to evaluate compliance with an SLO. For a Data Completeness SLO, the SLI is the specific calculation, such as (count of non-null records / total expected records) * 100. SLIs must be precisely defined, continuously measurable, and representative of the user experience of data quality.

An Error Budget is the allowable amount of unreliability, calculated as 100% - SLO. If a Data Completeness SLO is 99.9%, the error budget is 0.1%. This budget is a finite resource that quantifies the trade-off between innovation (releasing new features) and reliability. Consuming the budget triggers a focus on stability. It transforms SLOs from abstract targets into a management tool for engineering priorities.

A Data Freshness SLO defines the maximum acceptable age of data, specifying how long data can be delayed from its source event time before it is considered stale. It answers the question: "Is my data current enough?" Common SLIs measure the latency between an event's occurrence and its availability in the consuming dataset (e.g., p95 event time lag < 5 minutes). This is complementary to a completeness SLO, which ensures what is present is usable.

A Data Correctness SLO defines the maximum acceptable rate of inaccurate or invalid values within a dataset, based on business logic or validation rules. It answers: "Is my data accurate?" SLIs measure violations like:

• Values outside a plausible range (e.g., negative age).
• Referential integrity failures (e.g., orphaned foreign keys).
• Business rule violations (e.g., sum of line items ≠ total invoice). While completeness ensures presence, correctness ensures validity.

Automated Remediation is the practice of using software systems to automatically detect and resolve common failures in data pipelines without human intervention. For breaches of a Data Completeness SLO, examples include:

• Retrying failed ingestion jobs.
• Routing incomplete batches to a dead letter queue for analysis.
• Triggering alerts to on-call engineers only when automated steps fail. This practice directly reduces Mean Time to Resolution (MTTR) and operational toil.