Data Freshness SLO

The Service Level Indicator (SLI) is the specific, measurable metric used to quantify data freshness. This is the raw measurement of data age.

• Common Metrics: Maximum or P99/P95 latency from source event time to data availability in the destination table or API.
• Example: "The 99th percentile (P99) latency for user activity events to appear in the analytics.events table is less than 5 minutes."
• Measurement Point: Must be defined precisely—is it measured when data lands in the data warehouse, when a derived table is updated, or when an API consumer can query it?

This pair defines the objective itself: how good the freshness must be and over what period it is evaluated.

• Target Threshold: The numerical bound for the SLI (e.g., "< 5 minutes", "≥ 99.5%").
• Time Window: The rolling period over which compliance is calculated (e.g., 28 days, 30 days). A 30-day window is common, balancing responsiveness with statistical significance.
• Combined Example: "Over any rolling 30-day period, the P99 data freshness for the customer table must be less than 1 hour." This creates a stable, long-term target that prevents gaming of short-term metrics.

A Data Freshness SLO must be explicitly scoped to a specific data product or asset. A vague SLO is unactionable.

• Scope Definition: Clearly identifies the dataset, table, materialized view, or API endpoint. (e.g., prod.analytics.daily_sales_report, /api/v1/user-recommendations).
• Consumer Alignment: The scope should align with a known consumer or business process. Freshness for a real-time dashboard requires a stricter SLO than for a weekly batch report.
• Exclusions: May explicitly exclude known maintenance windows, backfill operations, or source system outages outside the team's control, if documented in the associated Error Budget Policy.

The Error Budget quantifies allowable unreliability and the policy dictates how it is managed.

• Budget Calculation: If the SLO is 99.5% freshness over 30 days, the error budget is 0.5% of that period: ~3.6 hours of allowed staleness.
• Budget Consumption: Each minute the data is stale beyond the SLO threshold consumes a portion of this budget.
• Policy Actions: A formal Error Budget Policy defines organizational responses. Examples:
  • Budget > 50% Remaining: Normal feature development and deployments proceed.
  • Budget < 25% Remaining: A freeze on risky changes; focus shifts to reliability work.
  • Budget Depleted: All non-critical work stops; full-team focus on restoring freshness.

The technical implementation for continuously measuring the SLI. Without accurate measurement, the SLO is theoretical.

• Event Time Capture: Requires pipelines to propagate a reliable event timestamp or processing timestamp from the source system.
• Monitoring System: A time-series database (e.g., Prometheus) or data observability platform that computes the freshness SLI in real-time.
• Dashboards & Alerts: Real-time dashboards show current freshness and budget burn. Alerts are triggered based on Burn Rate (speed of budget consumption) rather than single threshold breaches to reduce alert fatigue.
• Example Query: max_over_time((current_time - event_timestamp)[30d:5m]) to track maximum observed latency.

The process that ensures the SLO reflects actual business needs and evolves over time. It connects technical metrics to business value.

• Stakeholder Alignment: The SLO threshold and scope should be negotiated with and agreed upon by the primary consumers of the data (e.g., analytics, machine learning, or product teams).
• Periodic Review: SLOs should be reviewed quarterly or biannually. Questions include:
  • Is the current threshold still correct for consumer needs?
  • Is the error budget being consumed too fast or too slow?
  • Should the SLO be tightened, loosened, or retired?
• Documentation: The agreed-upon SLO, its rationale, and review dates are documented in a central service catalog or runbook.

A Service Level Objective (SLO) is a quantitative, internal target that defines the acceptable level of reliability for a specific service metric, such as availability or latency, over a defined period. It is the cornerstone of site reliability engineering (SRE) and data reliability engineering (DRE).

• Internal Goal: Unlike an SLA, an SLO is an internal target, often set more aggressively than the customer-facing guarantee.
• Basis for Decisions: SLOs drive engineering priorities, such as when to halt feature development to focus on stability.
• Example: "99.9% of daily active user records must be available for querying."

A Service Level Indicator (SLI) is a quantitative measure of a specific aspect of a service's performance, such as request latency or error rate, which is used to evaluate compliance with a Service Level Objective (SLO). For data, an SLI is the direct measurement of a quality dimension.

• Measured Metric: It is the raw measurement, like (records_on_time / total_expected_records) * 100.
• Feeds the SLO: The SLI value is compared against the SLO threshold to determine if the objective is being met.
• Example: For a Data Freshness SLO, the SLI could be the p95 age in minutes of all data records in a table.

An Error Budget is the allowable amount of unreliability, calculated as 100% - SLO, which provides a quantified resource for balancing the pace of innovation with the need for system stability.

• Trade-off Tool: It explicitly quantifies how much "unreliability" a team can afford to spend on launching new features versus fixing bugs.
• Consumption Rate: The Burn Rate measures how quickly the error budget is being consumed during an incident.
• Data Context: A Data Error Budget allows teams to make informed decisions about shipping a pipeline change that might slightly degrade freshness, versus pausing to improve reliability.

A Data Correctness SLO is a Service Level Objective that defines the maximum acceptable rate of inaccurate or invalid values within a dataset, based on business logic or validation rules. It complements a Freshness SLO to ensure data is both timely and accurate.

• Validation-Based: Often measured by the pass rate of automated data quality checks (e.g., (valid_records / total_records) * 100).
• Business-Critical: Directly impacts decision-making and model performance; incorrect data can be more damaging than stale data.
• Example: "Less than 0.1% of transaction records may contain an invalid currency code."

A Data Completeness SLO is a Service Level Objective that defines the minimum acceptable percentage of expected data records or fields that must be present and non-null for the dataset to be considered usable.

• Volume/Null Checks: Measures against an expected count or schema to detect missing data.
• Downstream Impact: Incomplete data can break joins, skew aggregates, and cause runtime errors in consuming applications.
• Example: "At least 99.5% of expected hourly user sessions must be recorded in the fact table."

Mean Time to Detection (MTTD) is a reliability metric that measures the average elapsed time between the onset of a system failure or incident and the moment it is first detected by monitoring systems or engineers. In data pipelines, low MTTD for freshness breaches is critical.

• Observability Link: Directly influenced by the quality of monitoring, alerting, and Data SLI measurement.
• Reducing Impact: A low MTTD minimizes the "blast radius" of bad data, preventing it from propagating to many downstream consumers.
• Complement to MTTR: Works with Mean Time to Resolution (MTTR) to form a complete view of incident lifecycle duration.