Data Lineage

This is the foundational layer responsible for automatically collecting lineage metadata. It involves instrumenting data pipelines to emit events at key points.

• Provenance Tracking: Logs the origin (source, timestamp, creator) of each data asset.
• Transformation Logging: Records the specific operations (joins, filters, aggregations) applied to data, often by hooking into execution engines like Apache Spark or dbt.
• Metadata Hooks: Integrations with tools like OpenLineage provide a standard schema for lineage events, which are sent to a collector.

The captured metadata is stored in a queryable database, typically modeled as a directed graph where nodes are data assets (tables, files, streams) and edges are processes or transformations.

• Graph Database: Technologies like Neo4j or JanusGraph are ideal for storing and traversing complex parent-child relationships.
• Asset Catalog Integration: The lineage store is often coupled with a data catalog (e.g., Apache Atlas, Amundsen) to enrich nodes with business context, schemas, and ownership.
• Temporal Versioning: Stores historical versions of the graph to answer "what was the lineage at this point in time?" for debugging past issues.

This is the primary analytical function powered by the lineage graph. It answers critical operational questions.

• Downstream Impact: Identifies all dashboards, models, and reports that depend on a specific upstream dataset. Crucial for assessing the blast radius of a schema change or data quality issue.
• Upstream Root Cause: Traces backwards from an erroneous output to pinpoint the exact source table, transformation step, or job that introduced the problem. This turns days of manual debugging into a minutes-long query.

A UI layer that renders the complex lineage graph into an interpretable format for engineers, analysts, and data stewards.

• Interactive Graph UI: Allows users to zoom, pan, and expand/collapse nodes to navigate dependencies.
• Column-Level Lineage: The gold standard, showing flow at the granularity of individual columns within a table, not just tables as a whole. This is essential for debugging specific metric calculations.
• Filtering & Search: Enables finding assets by name, owner, or domain to quickly isolate relevant sub-graphs.

Uses lineage as evidence for regulatory compliance and internal data governance.

• Data Privacy (GDPR, CCPA): Tracks where PII (Personally Identifiable Information) flows, enabling automated discovery and deletion requests ("right to be forgotten").
• Provenance for AI/ML: Critical for model governance, providing an audit trail of the exact training data used for a model version, which is required by regulations like the EU AI Act.
• Audit Logs: Immutable records of who accessed what data and when, generated from lineage access events.

Lineage is not isolated; it feeds and is fed by broader data platform observability tools.

• Pipeline Monitoring: When a data quality test fails, lineage instantly identifies downstream consumers to alert.
• Cost Attribution: By understanding which jobs produce which datasets, cloud compute costs (e.g., from BigQuery or Snowflake) can be allocated to business domains.
• CI/CD for Data: Lineage graphs are used to perform "dry runs" of pipeline changes to validate no critical downstream dependencies are broken before deployment.

Data provenance refers to the detailed, historical record of the origin and custody of a specific data item. It is a subset of lineage focused on the who, when, and where of data creation and ownership.

• Key Focus: Authenticity, trust, and audit trails for regulatory compliance.
• Example: Tracking that a specific customer record originated from a web form submission on 2023-10-05, was entered by user ID jsmith, and was initially stored in the raw_contacts table.
• Contrast with Lineage: While lineage maps the transformational journey, provenance certifies the original source and custodianship.

A data catalog is a centralized metadata repository that acts as an inventory of data assets. It provides context through business glossaries, data dictionaries, and ownership information, often integrating lineage for impact analysis.

• Primary Function: Data discovery and understanding for analysts and scientists.
• Integration Point: Modern catalogs ingest lineage graphs to answer questions like "Which reports use this column?" or "What is the source of this aggregated metric?"
• Tools: Examples include Apache Atlas, Alation, Collibra, and AWS Glue Data Catalog.

Data observability is the measure of the health and state of data in motion and at rest. It extends traditional monitoring by applying software engineering observability principles (metrics, logs, traces) to data pipelines.

• Five Pillars: Freshness, distribution, volume, schema, and lineage.
• Lineage's Role: Provides the topological map for root-cause analysis. When a dashboard metric breaks, lineage traces the failure upstream to the exact transformation or source system error.
• Goal: To detect, troubleshoot, and prevent data incidents before they impact downstream consumers.

Data drift is a change in the statistical properties (distribution, schema, semantics) of production data compared to the data a machine learning model was trained on, leading to degraded model performance.

• Lineage Connection: Lineage is critical for diagnosing drift. It identifies all upstream data sources and transformations that feed a model. When drift is detected, engineers use lineage to audit the contributing datasets and processes.
• Types: Covariate shift (input data changes), concept drift (relationship between input and output changes), and schema drift.

A data contract is a formal agreement between a data producer and consumer that specifies the schema, semantics, freshness, and quality guarantees of a data product.

• Enforcement Mechanism: Lineage systems can validate contracts by monitoring the pipelines that generate the data product, ensuring SLAs for latency and quality are met.
• Breach Detection: If a transformation violates the contracted schema (e.g., changes a column type), lineage can trigger an alert and identify the responsible job.
• Evolution: Contracts manage schema evolution, and lineage tracks which consumers are impacted by a change.

Impact analysis is the process of determining the downstream consequences of a proposed change to a data asset, such as a schema modification, pipeline failure, or source system decommission.

• Core Use Case for Lineage: This is a primary application of lineage graphs. By traversing the graph downstream from a given table, column, or job, engineers can generate a list of all dependent dashboards, reports, models, and applications.
• Business Value: Prevents breaking changes, enables accurate cost-benefit analysis for migrations, and streamlines communication during incidents.