Static Lineage

Static lineage is derived by parsing and analyzing source code and configuration artifacts. This includes:

• SQL scripts (SELECT, JOIN, CREATE TABLE statements)
• Workflow definitions (e.g., Airflow DAGs, dbt YAML files)
• Data transformation logic in tools like Spark or Dataform
• Infrastructure-as-Code templates (Terraform, CloudFormation)

The system builds a dependency graph by statically analyzing these artifacts, inferring relationships like table_a → view_b → dashboard_c. It does not require the jobs to be executed, making it a pre-runtime analysis technique.

Because it operates on static code artifacts, static lineage generation is deterministic. The same set of source files will always produce the same lineage graph. This enables:

• Version-controlled lineage: Lineage can be tied directly to a specific Git commit hash.
• Pre-merge validation: Impact analysis can be performed in CI/CD pipelines before code is deployed.
• Audit trail compliance: Provides a clear, reproducible map for governance that is not subject to runtime variability.

This characteristic is crucial for regulatory compliance and establishing a single source of truth for data dependencies that aligns with the software development lifecycle.

A primary limitation of static lineage is its inability to capture dynamic or runtime behavior. It cannot observe:

• Actual data flow volumes or execution timing.
• Runtime parameterization (e.g., a job that processes different partitions based on a config variable).
• Conditional logic branches that are data-dependent.
• Data quality rule violations or schema drift that occurs during execution.
• Implicit dependencies loaded via environment variables or secret managers.

This makes static lineage an incomplete picture for operational debugging but a perfect map for architectural understanding and impact analysis.

For explicitly declared dependencies in code, static lineage achieves very high fidelity. It excels at mapping:

• Column-level lineage by parsing SELECT clauses and column references in SQL.
• Transformation logic embedded within view definitions or model files.
• Upstream/downstream relationships between tables, models, and reports defined in orchestration tools.

Tools like SQL parsers (e.g., using ANTLR grammars) and abstract syntax tree (AST) traversal are used to extract these relationships with precision. The result is a highly accurate map of designed data flow, which is essential for documentation, governance, and planning.

Static lineage is the primary enabler for reliable impact analysis. By providing a complete graph of code-level dependencies, it allows teams to answer critical questions before making changes:

• What will break? Identify all downstream dashboards, models, and applications that depend on a table or column slated for modification.
• Who needs to be notified? Determine data consumers and asset owners affected by a proposed schema change.
• What is the test scope? Understand the full set of transformations that must be validated after a logic update.

This proactive analysis reduces operational risk and is a cornerstone of data reliability engineering practices.

Static lineage is fundamentally different from dynamic lineage, which is captured via runtime instrumentation. Key contrasts:

A robust data observability platform integrates both to provide a complete lineage picture: static for the 'blueprint' and dynamic for the 'as-built' record.

Dynamic lineage is captured at runtime by instrumenting the execution of data jobs, providing an accurate, operational record of what actually occurred. Unlike static analysis, it captures:

• Runtime parameters and configuration values used.
• Actual data volumes processed and rows affected.
• Execution timestamps and job status (success/failure).
• System resource consumption during the run. This lineage is essential for debugging specific job failures, auditing compliance, and understanding real resource usage, but it requires the pipeline to be executed to be generated.

Lineage granularity refers to the level of detail at which data flow is tracked. It exists on a spectrum:

• Job/Table-Level: Tracks dependencies between entire jobs or tables (coarse).
• Column-Level: Tracks the flow and transformation of individual data columns from source to destination (fine-grained).
• Row/Cell-Level: Tracks the provenance of individual data records or cells (extremely fine-grained). Static lineage tools typically aim for column-level granularity by parsing SQL SELECT and JOIN statements. Higher granularity provides more precise impact analysis but requires more sophisticated parsing and storage.

A dependency graph is a directed graph (often a Directed Acyclic Graph or DAG) that visually models the relationships between data assets, jobs, and pipelines. It is the primary output of lineage analysis. Key characteristics:

• Nodes represent data assets (tables, files, reports) or processing jobs.
• Edges represent dependencies ("depends on," "produces").
• Directed edges show the flow of data.
• Acyclic nature prevents infinite loops in pipeline scheduling. This graph enables impact analysis (tracing forward from a source) and root cause analysis (tracing backward from a failure). Static lineage is one method for automatically constructing this graph.

Lineage harvesting is the automated process of extracting lineage metadata from various sources within a data ecosystem. Static lineage is a primary harvesting method. Sources include:

• SQL Parsers: Analyze scripts in warehouses (BigQuery, Snowflake, Redshift).
• Orchestrator Metadata: Parse DAG definitions from tools like Apache Airflow, Dagster, or Prefect.
• ETL/ELT Tool Configs: Read transformation logic from dbt models, Informatica mappings, or Fivetran syncs.
• Business Intelligence Tools: Extract query logs from Looker, Tableau, or Power BI. The goal is to aggregate these disparate signals into a unified, centralized lineage graph without manual documentation.

Data provenance is a specific aspect of lineage focused on the origin and creation history of a data asset. It answers "where did this data come from and how was it created?"

• Provenance is a subset of lineage, often concerned with the initial sources and key transformations.
• It establishes authenticity and trustworthiness, crucial for regulated industries.
• While static lineage can infer provenance from code, dynamic lineage can provide a verifiable execution record. Provenance is key for debugging ("which raw source produced this erroneous output?") and for compliance with regulations like GDPR, which requires understanding data origins.