Lineage Granularity

Job-level lineage tracks dependencies between entire data processing jobs or pipeline tasks. It answers the question: 'Which jobs ran and in what order?'

• Scope: Maps coarse-grained dependencies (e.g., Spark Job A → Airflow DAG → Table B).
• Primary Use: High-impact analysis for pipeline failures, scheduling dependencies, and resource planning.
• Limitation: Cannot identify which specific columns or rows were affected by a job failure.
• Example: Knowing that a nightly customer_etl job feeds a daily_sales_report dashboard.

Dataset or table-level lineage tracks the flow of data between entire tables, files, or other storage entities. It answers: 'Which tables are the source and target of this data?'

• Scope: Shows relationships like raw_orders table → cleaned_orders table → orders_mart table.
• Primary Use: Understanding data provenance at the asset level, basic impact analysis for schema changes, and data cataloging.
• Common Implementation: Foundational level for most data catalogs and governance tools.
• Example: Tracing a BI chart back to its source fact and dimension tables in the data warehouse.

Column-level lineage tracks the flow and transformation of individual data columns. It answers: 'How was this specific column calculated, and what source columns contributed to it?'

• Scope: High-fidelity mapping showing, for example, that revenue_usd = quantity * unit_price_usd, where unit_price_usd is converted from unit_price_eur.
• Primary Use: Precise impact analysis for schema changes, debugging calculation errors, and validating business logic. Essential for GDPR 'right to erasure' or other data regulation compliance.
• Technical Challenge: Requires parsing SQL SELECT statements, stored procedures, and transformation code.

Row-level lineage tracks the journey of individual rows of data. It answers: 'What is the complete origin and transformation history for this specific record?'

• Scope: Follows a single customer record from an OLTP system, through various joins and filters, to its final state in a data mart.
• Primary Use: Critical for auditing, data forensics, and debugging issues affecting specific entities (e.g., 'Why is customer ID 12345 missing?').
• Performance Consideration: Capturing this granularity often requires significant runtime instrumentation and storage overhead.

Cell-level lineage represents the finest possible granularity, tracking the provenance of a single value within a table. It answers: 'What exact source data and operations produced this specific cell's value?'

• Scope: The ultimate form of data traceability. For example, tracing the $152.43 in a sales report cell back to the individual transaction, currency conversion rate, and tax calculation applied.
• Primary Use: Highest-stakes audit compliance, financial reporting validation, and root-cause analysis for microscopic data errors.
• Implementation: Extremely complex to capture at scale without prohibitive performance cost; often implemented selectively for critical data elements.

Selecting lineage granularity is an engineering trade-off between fidelity, performance, and cost.

• Coarse Granularity (Job/Table): Lower overhead, easier to implement, sufficient for architectural understanding and high-level impact analysis.
• Fine Granularity (Column/Row/Cell): Higher operational and storage cost, but enables precise debugging, granular governance, and regulatory compliance.

Best Practice: Implement a tiered approach. Capture high-level lineage everywhere, and enable fine-grained lineage for critical business data domains or regulated data subjects. The lineage fidelity of a system is determined by its weakest tracked granularity point.

Column-level lineage is a high-fidelity form of data lineage that tracks the flow and transformation of individual data columns from source to destination. It provides the detail necessary for precise impact analysis and debugging.

• Key Use Case: Identifying which downstream reports or models will break if a specific column's calculation logic changes.
• Technical Implementation: Typically requires parsing SQL SELECT statements, stored procedures, and transformation code to map column-to-column relationships.
• Contrast with Job-Level: Far more granular than job-level lineage, which only shows that Table A feeds into Job B, without detailing which columns are involved.

These are two primary methods for capturing lineage, differing fundamentally in when and how metadata is collected.

• Static Lineage: Derived by analyzing source code, SQL scripts, and pipeline definitions without execution. It shows intended dependencies.

  • Pros: Fast, cheap to generate, works for planned changes.
  • Cons: May miss runtime logic (e.g., conditional branches, parameterized queries).

• Dynamic Lineage: Captured by instrumenting jobs during runtime. It records what actually happened.

  • Pros: High accuracy, includes runtime parameters, data volumes, and execution context.
  • Cons: Requires pipeline instrumentation; cannot show lineage for unexecuted code.

Lineage fidelity measures the accuracy, completeness, and timeliness of captured lineage information. It answers the question: "How well does this lineage graph reflect the true, operational data flow?"

• High-Fidelity Lineage: Accurately maps all dependencies, including complex transformations and runtime behaviors. Essential for reliable root cause analysis.
• Low-Fidelity Lineage: May contain gaps, assumptions, or stale information, leading to lineage breaks that undermine trust.
• Drivers of Fidelity: Granularity (column vs. table), capture method (dynamic vs. static), and the breadth of integrated systems.

These are the primary operational use cases enabled by lineage granularity.

• Impact Analysis (Forward Tracing): The process of identifying all downstream dependencies (datasets, reports, models) that consume a given data asset. High granularity allows this analysis at the column level, providing a precise blast radius for changes.
• Root Cause Analysis (Backward Tracing): The systematic process of tracing a data quality issue or pipeline failure backward through the lineage graph to find the upstream source of the problem. Fine-grained lineage is critical to quickly isolate the faulty transformation or source column.

Data traceability is the overarching capability to follow the life of a specific data record both forwards and backwards through all processes and transformations. It is the ultimate business outcome of implementing high-granularity lineage.

• Forward Traceability: "Where did this customer record end up?" (e.g., in which models, reports, decisions).
• Backward Traceability: "What is the complete origin and transformation history of this calculated revenue figure?"
• Regulatory Driver: Essential for compliance with regulations like GDPR (Right to Explanation) and financial auditing, where the provenance and journey of specific data points must be documented.