OpenLineage

OpenLineage defines a vendor-neutral, extensible JSON schema for representing lineage metadata. This schema standardizes core entities:

• Runs: A single execution instance of a job.
• Jobs: A logical task or computation (e.g., a Spark job, dbt model).
• Datasets: Named collections of data with a defined schema.
• Facets: Extensible key-value objects that attach additional context (e.g., column-level lineage, data quality metrics, ownership). This standardization enables disparate tools to produce and consume lineage in a consistent format, breaking down metadata silos.

OpenLineage captures dynamic lineage by instrumenting data processing frameworks at runtime. Integrations (collectors) hook into execution engines like Apache Spark, Apache Airflow, dbt, and Great Expectations to emit lineage events as jobs run. This provides an accurate, factual record of:

• Actual data inputs and outputs, including file paths and table names.
• Transformation logic executed.
• Runtime context like execution time, status (start, complete, fail), and data volumes. Unlike static code analysis, runtime capture reflects what actually happened, accounting for conditional logic and runtime parameters.

The core schema is extended via the facet system, which allows attaching domain-specific metadata to runs, jobs, and datasets. Facets are the mechanism for capturing detailed context:

• Column-level lineage: Tracks the flow of individual columns through SQL queries or transformations.
• Data Quality assertions: Attach results from validation frameworks.
• Ownership and governance: Link datasets to teams, PII classifications, or data contracts.
• Custom facets: Organizations can define proprietary facets for internal metadata. This extensibility makes OpenLineage a foundational layer for a unified metadata ecosystem.

OpenLineage employs a decoupled architecture where lineage producers (instrumented applications) emit events, and lineage consumers (backends) collect, store, and serve them. Events are typically sent via an HTTP API or message queue (e.g., Kafka). This separation provides critical flexibility:

• Producers need no knowledge of the storage backend.
• Organizations can choose or build their own backend (e.g., Marquez, Datakin, OpenMetadata, custom data lakes).
• Enables centralized collection of lineage from heterogeneous sources across the entire data stack.

As a Linux Foundation project, OpenLineage fosters a broad ecosystem of pre-built integrations. This reduces the effort to instrument diverse technologies:

• Orchestrators: Airflow, Dagster, Prefect.
• Processing Engines: Spark, Flink, Beam.
• Transformation Tools: dbt, SQL-based warehouses (BigQuery, Snowflake, Redshift).
• Quality Tools: Great Expectations. This growing ecosystem allows organizations to assemble an end-to-end, cross-system lineage view without being locked into a single vendor's platform.

The captured lineage metadata serves as the backbone for advanced data observability and governance use cases:

• Impact Analysis: Identify all downstream dashboards and models affected by a schema change or pipeline failure.
• Root Cause Analysis (RCA): Trace a data quality issue backward through the lineage graph to find the source.
• Data Freshness & SLA Monitoring: Correlate job run events with dataset update times.
• Compliance & Auditing: Provide a verifiable audit trail of data provenance and transformations for regulatory requirements. By providing a factual graph of data flow, OpenLineage turns lineage from a static diagram into an operational asset.

Data lineage is the comprehensive record of the origin, movement, transformation, and dependencies of data across its lifecycle. It provides an audit trail used for governance, debugging, and impact analysis. OpenLineage is a standard for capturing this metadata.

• Purpose: Enables root cause analysis, compliance reporting, and change management.
• Scope: Can be tracked at the dataset, column, or even cell level.

Data provenance is a specific aspect of lineage that documents the origin and creation history of a data asset. It establishes authenticity and trustworthiness by answering where data came from and how it was produced.

• Focus: Origin and derivation, rather than full end-to-end flow.
• Use Case: Critical for regulatory compliance (e.g., GDPR) and reproducing scientific or analytical results.

Dynamic lineage is captured at runtime by instrumenting the execution of data jobs. It provides an accurate, operational record of what actually occurred, including runtime parameters, data volumes, and execution status.

• Contrast with Static Lineage: Derived from live execution, not static code analysis.
• OpenLineage Role: The OpenLineage specification is designed to capture dynamic lineage events from running pipelines.

A dependency graph is a directed graph that visually models the relationships and dependencies between data assets, jobs, and pipelines. It is the primary structure used to represent lineage.

• Structure: Typically a Directed Acyclic Graph (DAG).
• Function: Enables automated impact analysis (what breaks if this changes?) and root cause analysis (what caused this data issue?).

Impact analysis is the process of identifying all downstream data assets, reports, and models that depend on a given data source or transformation. This is performed by traversing the lineage dependency graph.

• Proactive Use: Assessing the scope of a proposed schema change or pipeline update.
• Reactive Use: Understanding the blast radius of a data quality incident or pipeline failure.

Root cause analysis is the systematic process of tracing data quality issues or pipeline failures backward through the lineage graph to identify the original source of the problem.

• Process: Starts with an erroneous output and follows upstream dependencies to find the broken transformation or corrupted source.
• Requirement: Depends on high-fidelity, accurate lineage to be effective.