Data Lineage

The ability to trace data from its point of consumption back to its original source systems. This answers the question, 'Where did this data come from?'

• Identifies every upstream system, table, and transformation
• Critical for debugging data quality issues in reports
• Enables rapid root cause analysis when anomalies are detected
• Example: Tracing a financial report figure back through a data warehouse to the raw transaction log

The ability to identify all downstream consumers and assets that will be affected by a change to a source field or pipeline. This answers, 'Who depends on this data?'

• Maps dependencies to dashboards, ML models, and APIs
• Prevents breaking changes during schema evolution
• Allows data engineers to communicate change deprecation safely
• Example: Identifying every executive dashboard that will break if a column is renamed

Lineage must operate at the column or field level, not just the table or dataset level. This provides surgical precision for impact analysis and debugging.

• Tracks how a single field like customer_id flows through joins and aggregations
• Distinguishes between direct pass-through and derived calculations
• Essential for compliance with regulations like GDPR for tracking PII
• Example: Knowing that revenue in a report is calculated from price * quantity - discount

Lineage is not just a map of connections; it must capture the actual business logic applied at each step. This provides semantic understanding, not just structural mapping.

• Records SQL queries, Python scripts, or dbt models applied
• Documents filtering conditions, aggregations, and business rules
• Allows analysts to understand how data was altered, not just where it moved
• Example: Capturing that a status field was filtered to only include 'active' records before aggregation

Manual lineage mapping is brittle and quickly becomes outdated. Robust systems use automated parsing of query logs, execution plans, and code repositories.

• Parses SQL from BI tools, ETL scripts, and data warehouses
• Integrates with transformation frameworks like dbt for native lineage
• Updates lineage graphs in near real-time as pipelines change
• Example: Automatically discovering a new dependency when an analyst creates a Tableau data source

Data rarely stays in one system. True lineage stitches together flows across heterogeneous environments: databases, object storage, streaming platforms, and APIs.

• Connects lineage from ingestion (Kafka) to storage (S3) to serving (Snowflake)
• Handles both batch and streaming pipelines uniformly
• Prevents blind spots at system boundaries where data quality often degrades
• Example: Tracking an IoT event from an MQTT broker through a stream processor into a feature store

A specialized subset of lineage that focuses specifically on the origin and ownership of data. While lineage tracks the entire journey, provenance answers 'Where did this data originally come from?' and 'Who created it?'.

• Captures input sources, sensors, or human creators
• Documents the chain of custody for auditability
• Essential for scientific reproducibility and regulatory compliance
• Often implemented via W3C PROV standard

The continuous monitoring of data health across pipelines, encompassing freshness, volume, schema, and lineage. Observability platforms operationalize lineage graphs to detect anomalies and trigger alerts when data deviates from expected norms.

• Monitors five pillars: freshness, distribution, volume, schema, lineage
• Uses lineage to perform rapid root cause analysis on data incidents
• Integrates with tools like Monte Carlo, Datadog, and Great Expectations

The systematic administration of data that describes other data. Lineage is a critical form of structural metadata that maps relationships between datasets, transformations, and consumers.

• Powered by data catalogs like Alation, Collibra, and Atlan
• Enables impact analysis before making schema changes
• Feeds into data governance dashboards for compliance officers

The mathematical structure underlying most lineage representations. A DAG models data transformations as nodes connected by directed edges, ensuring no circular dependencies exist in the pipeline.

• Used by orchestrators like Apache Airflow and Dagster
• Enables task dependency resolution and parallel execution
• Visualized as upstream-downstream lineage graphs in data platforms

The overarching framework of policies, standards, and accountability that ensures data is managed as a strategic asset. Lineage provides the technical enforcement mechanism for governance policies.

• Supports GDPR Article 30 record-keeping requirements
• Enables data classification and access control propagation
• Bridges the gap between legal compliance and engineering execution

The highest resolution of lineage tracking, mapping transformations at the individual field or attribute level rather than entire tables. Critical for understanding exactly how a specific metric is calculated.

• Traces a single column through joins, aggregations, and casts
• Essential for regulatory reporting where field-level accuracy is audited
• Supported by tools like dbt Explorer and SQLGlot parsers