Data Lineage Graph

The foundational data structure for lineage, where nodes represent datasets or transformations and directed edges define the flow of data. The acyclic property prevents circular dependencies, ensuring a clear, deterministic path from origin to output. This structure enables topological sorting for replaying pipelines and identifying upstream bottlenecks. Unlike simple linear tracking, a DAG captures complex fan-in (multiple sources merging) and fan-out (one source feeding multiple consumers) patterns common in feature engineering.

Moves beyond table-level tracking to map the specific transformations applied to individual fields. This is critical for Personally Identifiable Information (PII) compliance and debugging. For example, it traces how a raw user_id column is hashed, joined with a transactions table, and ultimately feeds a credit_score feature in a model. This granularity allows engineers to instantly assess the blast radius of a schema change or a data quality anomaly in a single column.

Lineage is not static; it evolves with code and data. A robust graph captures time-travel capabilities by versioning both the data assets and the transformation logic (e.g., SQL queries, Python scripts). This allows users to query the state of the lineage graph at a specific point in time, enabling historical debugging. It answers the question: 'What did the pipeline look like when this erroneous batch was generated last Tuesday?'

Manual lineage mapping is brittle and outdated instantly. Modern systems use static code analysis to parse SQL, Python (e.g., Pandas, Spark), and dbt models to automatically construct the graph. They integrate with orchestrators like Airflow and catalog tools via APIs. This automation captures implicit dependencies that human documentation misses, ensuring the graph remains a faithful, real-time digital twin of the actual execution environment.

The primary operational interface for the graph. Forward lineage (impact analysis) predicts downstream effects: 'If I deprecate this API field, which dashboards and models break?' Backward lineage (root-cause analysis) traces errors upstream: 'Why is the executive KPI dashboard showing null values?' The engine traverses the DAG to instantly visualize the propagation path of changes or errors, reducing incident response from hours to minutes.

To avoid vendor lock-in, enterprise lineage graphs often serialize data using the W3C PROV-O ontology or OpenLineage standards. This provides a standardized JSON schema for representing runs, jobs, and datasets. Interoperability allows the lineage graph to federate queries across disparate systems—connecting a Spark job in Databricks to a dbt transformation in Snowflake—creating a unified, cross-platform map of the entire data estate.

The documented chronology of data origin, transformations, and custody that establishes a verifiable chain of ownership and integrity for digital assets. Unlike lineage graphs which focus on technical transformations, provenance captures the broader who, what, when, and why of data handling. Essential for regulatory compliance under frameworks like GDPR and the EU AI Act, provenance records provide the evidentiary backbone for audits and impact assessments.

The W3C PROV Ontology is a formal OWL2 specification for representing and interchanging provenance information across heterogeneous systems. It defines three core classes:

• Entity: Physical or digital assets
• Activity: Processes that transform entities
• Agent: Responsible actors or software This standardized data model enables interoperable lineage graphs that can span multiple tools and organizational boundaries, making it the lingua franca for enterprise provenance exchange.

A chronologically ordered, write-once-read-many (WORM) log of all events and transactions related to a data asset. Cryptographically secured using Merkle tree verification to prevent retroactive alteration, immutable audit trails provide the definitive historical record that lineage graphs visualize. Each entry is hashed and chained, ensuring that any tampering is immediately detectable. This is the foundational data structure that makes lineage graphs legally defensible.

A framework and metadata format for cryptographically signing and verifying each step in a software supply chain. In-toto produces a verifiable, end-to-end provenance trail from source code to final deployment by collecting signed attestations at every pipeline stage. When applied to data pipelines, it creates cryptographically verifiable lineage edges that prove a specific transformation was executed by an authorized process with a known binary, preventing supply chain attacks on data infrastructure.

A family of World Wide Web Consortium specifications defining a standardized data model, ontology, and serializations for representing provenance. Key components include:

• PROV-DM: The conceptual data model
• PROV-O: The OWL2 ontology for RDF
• PROV-N: A human-readable notation Together, these specifications ensure that lineage graphs can be serialized, queried, and federated across different platforms using a common semantic framework.

A chronological, auditable documentation trail that records the sequence of entities who have held, transferred, or modified a specific data asset. While a lineage graph maps technical transformations, chain of custody maps legal and physical possession. In regulated industries like healthcare and finance, maintaining an unbroken chain of custody is essential for ensuring data integrity and admissibility in legal proceedings. It answers the question: 'Who had control of this data at each point in time?'