Data Lineage

The most precise form of lineage traces data at the column level, mapping how a single field in a report is derived from specific source columns through a chain of transformations. This contrasts with table-level lineage, which only tracks whole datasets.

• Example: A customer_lifetime_value field in a dashboard is traced back through a dbt model to orders.total and customers.acquisition_cost in the raw ingestion layer.
• Benefit: Enables precise impact analysis and regulatory compliance for sensitive fields like PII.

Modern lineage systems automatically extract relationships by parsing the code that transforms data, rather than relying on manual documentation. They analyze SQL queries, Python scripts, and ETL job configurations to build a dependency graph.

• SQL Parsing: Identifies source and target columns in SELECT, INSERT, and MERGE statements.
• OpenLineage Standard: An open framework where job schedulers like Apache Airflow emit lineage metadata as events to a central collector.
• Benefit: Eliminates the drift between documented and actual data flows.

A complete lineage graph stitches together flows across disparate systems—from streaming platforms like Apache Kafka to data warehouses like Snowflake and BI tools like Looker. It captures both batch and real-time pipelines.

• Cross-System Edges: A lineage tool must correlate a Kafka topic's schema to a Spark job's input and then to a dashboard's dataset.
• Benefit: Provides a single pane of glass for data observability, allowing engineers to trace an error in a report back to a failed sensor ingestion in seconds.

Lineage is not static; it must capture the state of data and its transformations at specific points in time. This is achieved through data versioning and table formats like Delta Lake or Apache Iceberg.

• Time Travel Queries: Allow users to query a table as it existed at a specific timestamp or transaction ID.
• Lineage Snapshots: The lineage graph itself must be versioned to show how transformation logic changed between releases.
• Benefit: Enables reproducibility for audits and debugging, allowing teams to see exactly what code produced a now-erroneous report.

Lineage is the engine for both forward-looking impact analysis and backward-looking root cause analysis. It answers 'What will break if I change this?' and 'What caused this error?'

• Forward Trace: Before deprecating a source column, an engineer can instantly see every downstream model, dashboard, and ML feature that depends on it.
• Backward Trace: When a CFO's report shows a null value, the lineage graph is traversed upstream to identify the exact transformation step that introduced the null.
• Benefit: Reduces incident resolution time from hours to minutes.

Lineage enforces data contracts by making dependencies explicit. A contract defines a producer's schema and quality guarantees; lineage monitors whether consumers are adhering to the contract and alerts on violations.

• Schema Enforcement: If a producer changes a column type, lineage identifies all downstream consumers that will break, triggering a contract violation alert before deployment.
• Ownership Mapping: Lineage graphs associate datasets with the owning teams, clarifying who is responsible for fixing a pipeline failure.
• Benefit: Prevents the silent breakage of critical data products in a decentralized data mesh.

A World Wide Web Consortium standard defining a data model and serializations for representing and exchanging provenance information.

• Core model uses three classes: Entities (data), Activities (transformations), and Agents (responsible actors)
• Provides PROV-O (OWL2 ontology) and PROV-N (human-readable notation) serializations
• Enables interoperable lineage across heterogeneous tools that adopt the standard
• Forms the conceptual foundation for many enterprise lineage platforms

The process of assessing the downstream consequences of a proposed data change by tracing lineage forward to identify all dependent assets, reports, and models.

• Answers: "If I deprecate this source column, which dashboards break?"
• Critical for change management in regulated environments (SOX, GDPR)
• Reverse of root cause analysis, which traces backward from an error to its origin
• Automated by lineage tools that maintain a directed acyclic graph (DAG) of dependencies

A chronological, tamper-proof record of all data access and modification events, ensuring non-repudiation and supporting forensic analysis.

• Built on append-only storage patterns where records cannot be deleted or altered
• Uses cryptographic hashing (Merkle trees) to verify integrity of the entire chain
• Required for compliance frameworks including HIPAA, SOC 2, and FedRAMP
• Distinct from lineage: audit trails capture who did what when, lineage captures where data came from

The practice of storing and managing unique snapshots of datasets at specific points in time, enabling reproducibility, rollback, and comparison of data states.

• Implemented via time travel in Delta Lake and Apache Iceberg table formats
• Uses transaction IDs or timestamps as version identifiers
• Enables reproducible ML training by pinning experiments to exact data versions
• Complements lineage by providing the what and when alongside the where from