Data SLO (Service Level Objective)

The Service Level Indicator (SLI) is the foundational, quantitative measure of performance for a specific data quality characteristic. It is the raw metric used to evaluate compliance with the SLO.

• Examples: Percentage of records delivered within a 1-hour freshness window; proportion of rows passing completeness checks; count of schema validation errors per million records.
• Key Property: An SLI must be a direct, measurable value, not a subjective assessment. It is the instrument reading that tells you the current state of your data service.

The Target Threshold is the specific numerical goal that defines the SLO. It is the line between acceptable and unacceptable performance for the SLI over a defined measurement period.

• Format: Typically expressed as a percentage or a maximum allowable value (e.g., "99.9% of records are fresh," "< 0.1% null values in key fields").
• Purpose: This threshold creates a binary, objective condition for success or failure, moving reliability from an abstract concept to an engineering target. Setting it requires balancing user expectations with engineering feasibility.

The Measurement Window is the rolling time period over which the SLI is calculated and compared against the target threshold. It defines the scope of evaluation for SLO compliance.

• Common Windows: 28-day or 30-day rolling windows are standard, aligning with monthly business cycles. Shorter windows (e.g., 7 days) provide more responsive feedback but can be noisy.
• Critical Function: This window determines the statistical significance of the measurement. A single hour of downtime has a vastly different impact on a 28-day SLO versus a 1-day SLO. It is essential for calculating the error budget.

The Error Budget is the explicit, allowable amount of unreliability derived directly from the SLO. It represents the "budget" of failure a data pipeline can spend before triggering formal review or remediation efforts.

• Calculation: If an SLO is 99.9% reliability over 30 days, the error budget is 0.1% of that period, or 43.2 minutes of allowable unavailability.
• Engineering Utility: This concept transforms SLOs from passive goals into active management tools. Teams can consciously "spend" their budget on risky deployments or innovation, and must halt feature work to focus on stability once the budget is exhausted.

The Data Asset Scope explicitly defines which dataset, table, pipeline, or data product the SLO applies to. Ambiguity here renders the SLO unactionable.

• Specification Must Include: The fully qualified name of the table (e.g., prod.analytics.fct_orders), the specific data pipeline job ID, or the logical data product identifier in a data mesh.
• Importance: Clear scoping prevents debates about responsibility when the SLO is breached and ensures monitoring instrumentation is applied to the correct entity. It links the reliability promise to a concrete, owned asset.

This component defines what aspect of data quality is being measured, framed from the consumer's perspective. It answers the question: "What does 'reliable data' mean for the user of this asset?"

• Core Characteristics:
  • Freshness/Timeliness: Data is up-to-date (e.g., SLI = time since last successful update).
  • Completeness: Required data is present (e.g., SLI = % of non-null values in critical columns).
  • Accuracy: Data correctly reflects reality (e.g., SLI = % of records matching a verified source).
  • Validity: Data conforms to a defined schema and business rules.
• Principle: The chosen characteristic must map directly to a user-visible symptom if it degrades. An analytics dashboard showing stale numbers is a freshness failure.

A Data Service Level Indicator (SLI) is the quantitative, measurable component that underpins a Data SLO. It is a direct measurement of a specific aspect of data service performance over a defined time window.

• Examples include the percentage of records delivered within a 1-hour freshness threshold, the proportion of rows passing completeness checks, or the rate of schema validation failures.
• The SLI provides the raw data; the SLO defines the target for that data. For instance, an SLO might state "99.9% of records must be fresh (<1 hour old)," while the corresponding SLI is the actual, measured percentage of fresh records.

A Data Error Budget is the explicit, allowable amount of unreliability derived from a Data SLO. It quantifies how much a data pipeline can "fail" its SLO before triggering formal review or remediation efforts.

• Calculation: If an SLO is set at 99.9% reliability over a 30-day period, the error budget is the remaining 0.1%, or approximately 43 minutes of allowable downtime/unreliability.
• Purpose: It transforms SLOs from abstract goals into a management tool. Teams can spend the budget on innovation (e.g., risky schema changes) but must halt non-essential work if the budget is exhausted to focus on stabilizing reliability.

Data Reliability Engineering (DRE) is the discipline of applying Site Reliability Engineering (SRE) principles specifically to data systems. It is the overarching practice that operationalizes Data SLOs and error budgets.

• Core Tenets: Defining, measuring, and achieving reliability for data products through systematic processes, automated monitoring, and blameless post-mortems.
• Key Activities: Establishing Data SLOs, implementing Data SLIs, managing error budgets, building automated remediation playbooks, and practicing Data Incident Management. DRE shifts the focus from reactive firefighting to proactive, engineered reliability.

Data Downtime is a period during which a dataset is incomplete, inaccurate, stale, or otherwise unfit for its intended use, making it analogous to application downtime in traditional software systems.

• Causes: Pipeline failures, source system outages, schema drift, or undetected data corruption.
• Measurement: Data Downtime is directly measured against Data SLOs. For example, if a freshness SLO is violated for 2 hours, that constitutes 2 hours of data downtime.
• Impact: It quantifies the business cost of unreliable data, affecting downstream analytics, machine learning models, and operational reports. Reducing data downtime is a primary goal of implementing SLOs.

Data Contract Monitoring is the automated enforcement of formal agreements between data producers and consumers. While an SLO is an internal reliability target, a data contract is an external promise about data characteristics.

• Content: Contracts specify guarantees on schema, semantics, freshness, and quality metrics.
• Relation to SLOs: A producer's internal Data SLOs (e.g., 99.9% freshness) are the mechanism to ensure they can reliably meet the commitments in their external data contracts. Monitoring validates that the contract's terms are continuously upheld, with breaches triggering alerts similar to SLO violations.

Automated Remediation refers to predefined, programmatic actions executed by an observability platform in response to specific, detected failures that threaten Data SLO compliance.

• Objective: To reduce Mean Time To Resolution (MTTR) for Data and preserve error budgets by resolving common issues without human intervention.
• Examples: Automatically retrying a failed ingestion job, switching to a backup data source when primary latency spikes, or quarantining a batch of data that fails validation rules.
• SLO Context: Effective remediation is triggered by alerts based on Data SLI degradation and is designed to restore service before an SLO is breached.