Data SLI

Measures the timeliness of data delivery. This SLI tracks the age of data from its source event time to its availability for consumption.

• Core Metric: Data Lag (e.g., p95 event_time - ingestion_time < 5 minutes).
• Example: "Percentage of dashboard queries executed on data less than 1 hour old."
• Impact: Stale data leads to incorrect business decisions and erodes trust in analytics.

Measures whether all expected data arrives. This SLI tracks missing records, files, or partitions against an expected volume or schedule.

• Core Metric: Record Count Ratio (e.g., (actual_records / expected_records) * 100).
• Example: "99.9% of daily user event files are delivered by 06:00 UTC."
• Impact: Missing data creates gaps in time-series analysis and skews aggregate metrics.

Measures the accuracy and adherence of data to defined schemas and business rules. This SLI tracks invalid values, schema drift, and referential integrity violations.

• Core Metric: Validation Failure Rate (e.g., (invalid_records / total_records) * 100).
• Example: "Less than 0.1% of records fail column type or range validation."
• Impact: Invalid data corrupts downstream models, reports, and operational systems.

Measures the accessibility and operational status of the data asset itself. This SLI tracks whether the data table, API, or file is queryable and online.

• Core Metric: Uptime Percentage (e.g., (successful_queries / total_queries) * 100).
• Example: "The customer dimension table is queryable 99.95% of the time."
• Impact: Unavailable data halts dependent pipelines, dashboards, and applications.

Measures unexpected fluctuations in data size. This SLI detects anomalies in row counts or data size that may indicate a processing error or source system change.

• Core Metric: Daily Volume Delta (e.g., ABS(today_count - avg_last_7_days) / avg_last_7_days).
• Example: "Daily transaction record count does not deviate by more than ±15% from the 30-day rolling average."
• Impact: Drastic volume changes can indicate broken source connectors or fraudulent activity.

Measures the health of upstream dependencies. This SLI tracks the successful execution and quality of parent datasets or pipelines in the data lineage graph.

• Core Metric: Upstream SLI Rollup (e.g., aggregate status of all direct parent jobs).
• Example: "100% of critical upstream source pipelines completed successfully."
• Impact: Failures cascade; a single broken source can invalidate dozens of downstream derived datasets.

A Service Level Objective (SLO) is the quantitative, internal target for a specific Service Level Indicator (SLI). It defines the acceptable level of reliability for a data product over a defined period.

• Example: "99% of daily user records must arrive within 15 minutes of the source event (Data Freshness SLO)."
• SLOs are derived from business requirements and user expectations.
• They create a clear, shared goal for data engineering and product teams.

A Data Error Budget is the allowable amount of unreliability for a data product, calculated as 100% - SLO. It quantifies the trade-off between innovation and stability.

• Example: A 99% freshness SLO permits a 1% error budget (e.g., ~7.2 hours of stale data per month).
• Consuming the budget signals quality issues; a depleted budget should trigger a focus on reliability work.
• It operationalizes SLOs by providing a clear, consumable resource for decision-making.

A Service Level Agreement (SLA) is a formal contract with external customers that defines the minimum acceptable service performance, often with financial penalties for breach.

• Contrast with SLO: An SLO is an internal target; an SLA is an external promise.
• SLAs are typically set less aggressively than internal SLOs to provide a safety buffer.
• For internal data platforms, SLAs may be established between producer and consumer teams.

A Data Freshness SLO is a specific type of Service Level Objective that defines the maximum acceptable age of data, measured from the source event time to data availability.

• Core SLI: (Records within freshness window) / (Total expected records).
• Example Objective: "95% of hourly sales data must be available for query within 5 minutes of the hour's end."
• Critical for time-sensitive use cases like real-time dashboards and operational alerts.

A Data Correctness SLO defines the maximum acceptable rate of inaccurate or invalid values within a dataset, as measured against business logic or validation rules.

• Core SLI: (Records passing validation checks) / (Total records processed).
• Example Objective: "The schema validation failure rate must be less than 0.1%."
• Validations can include data type checks, referential integrity, value range adherence, and custom business rules.

A Data Completeness SLO defines the minimum acceptable percentage of expected data records or fields that must be present and non-null for a dataset to be considered usable.

• Core SLI: (Records/fields received) / (Total records/fields expected).
• Example Objective: "Daily batch ingestion must achieve at least 99.9% record completeness."
• Often monitored by comparing source system record counts with destination counts or by tracking null rates in key columns.