Data Observability

Measures how up-to-date data tables are and whether they have been updated within expected time windows. Freshness monitoring tracks the lag between when data is generated and when it becomes available for consumption.

• Key metric: Time since last update
• Example: A financial dashboard expecting trade data every 15 minutes triggers an alert if the pipeline stalls for over 30 minutes
• Impact: Stale data in machine learning feature stores leads to degraded model predictions and incorrect business decisions

Assesses the statistical properties of data at the field level to detect unexpected shifts that may indicate broken pipelines or systemic errors. This pillar answers the question: does the data fall within an expected range?

• Key metric: Divergence from historical baseline (KL divergence, KS test)
• Example: A null rate on a customer_id column suddenly jumping from 0.1% to 15% signals an upstream ingestion failure
• Impact: Undetected distribution shifts are a primary cause of silent model failure in production

Monitors the completeness and quantity of data arriving in tables, ensuring that records are neither missing nor duplicated. Volume checks provide a coarse-grained signal of pipeline health.

• Key metric: Row count vs. historical average
• Example: A daily sales table that typically receives 2 million rows suddenly dropping to 200,000 indicates a failed extract-transform-load job
• Impact: Incomplete data volumes cause downstream reports and dashboards to present misleading business metrics

Tracks changes to the structure and organization of data, including added, removed, or type-changed columns. Schema observability is critical for preventing breaking changes in production pipelines.

• Key metric: Schema change events detected
• Example: A price column changing from DECIMAL(10,2) to VARCHAR breaks downstream aggregations and model inference
• Impact: Unannounced schema changes are the most common cause of data pipeline outages in complex microservice architectures

A change in the statistical properties or distribution of the input data fed to a machine learning model over time, which can degrade predictive performance. Observability platforms monitor for feature drift, label drift, and concept drift as key health metrics. For example, a fraud detection model trained on pre-pandemic transaction patterns may silently fail as consumer behavior shifts—a phenomenon detectable only through continuous distribution monitoring.

A formal, machine-readable agreement between a data producer and its consumers that defines the schema, semantics, and quality guarantees of the data being provided. Data contracts act as programmable policies within an observability framework, automatically triggering alerts when a producer violates its stated service-level objectives (SLOs) for freshness or schema stability. They shift quality enforcement left, preventing bad data from ever reaching downstream models.

A directed acyclic graph (DAG) that visually represents the historical dependencies, transformations, and origins of a data artifact. The provenance graph is the core data structure enabling root-cause analysis and impact assessment in observability platforms. When a field's value suddenly drops to null, engineers traverse the graph backward to identify the exact transformation step where the corruption occurred, dramatically reducing mean time to detection (MTTD).

A chronological, secure record of system activities and data access events that provides documentary evidence for reconstructing and examining the sequence of operations in a data lifecycle. In regulated industries, the audit trail complements observability by answering not just what changed in the data, but who made the change and when. This is critical for SOX compliance and GDPR data subject access requests.