Data Lineage

The level of detail at which lineage is recorded, ranging from coarse table-level tracking to fine-grained column-level and row-level provenance.

• Table-Level: Tracks movement and transformation of entire datasets. Simplest to implement but masks internal changes.
• Column-Level: Follows specific attributes through pipelines, critical for understanding how a sensitive field like PII was derived.
• Row-Level: Captures the exact input records that produced each output row, essential for debugging specific anomalies in ETL/ELT processes.

Lineage is mapped in two directions to serve different investigative needs.

• Backward Lineage: Traces data upstream to its origin. Answers "Where did this data come from?" Used for debugging errors found in a report.
• Forward Lineage: Tracks data downstream to its consumers. Answers "What is impacted if I change this field?" Used for impact analysis before deprecating a source column.
• End-to-End Lineage: Combines both to provide a full horizontal view from source systems to final consumption dashboards or models.

The mechanism of recording not just that data changed, but how it changed. This moves lineage from a static map to a dynamic audit tool.

• Implicit Capture: A parser automatically reads SQL queries, notebook code, or ETL tool configurations to extract transformation logic.
• Explicit Stitching: An API is used to manually declare inputs, outputs, and the specific function applied when automatic parsing is impossible.
• Capturing the exact WHERE clause or aggregation function is vital for regulatory compliance and reproducing historical results.

The ability to view lineage as it existed at a specific point in the past, not just the current state. Data pipelines evolve, and yesterday's truth may differ from today's.

• Point-in-Time Recovery: Reconstructs the exact lineage graph for a prior execution, enabling auditors to validate a financial report generated six months ago.
• Diff Analysis: Compares two versions of a lineage graph to instantly identify new, modified, or deleted transformations, accelerating root cause analysis for sudden data quality drops.

The process of carrying technical and business context alongside the data as it transforms. Lineage is incomplete without the metadata that gives data meaning.

• Technical Metadata: Schema definitions, data types, and record counts that flow through the pipeline to validate structural integrity.
• Business Metadata: Tags, data classifications (e.g., GDPR, PCI), and ownership that must persist to ensure downstream consumers apply correct access controls and usage policies.
• A break in metadata propagation is a primary indicator of a lineage gap.

The method by which lineage connections are created, directly impacting completeness and trust.

• Automated Discovery: Tools scan query logs, data catalogs, and execution engines to build lineage graphs with minimal human intervention. This is the only scalable approach for modern data mesh environments.
• Manual Mapping: Required for legacy systems or black-box processes where logic cannot be extracted. Often recorded in spreadsheets, this method is fragile and prone to staleness.
• A hybrid approach uses automated discovery for the majority and manual stitching to close critical gaps.