Data Provenance

A chronological, tamper-proof record of all data access and modification events. This ensures non-repudiation, meaning no party can deny their actions, and supports forensic analysis.

• Uses cryptographic hashing to detect alterations
• Critical for SOC 2 and GDPR compliance
• Provides a verifiable history for auditors

The World Wide Web Consortium (W3C) standard defines a data model for representing provenance. It structures information around three core types:

• Entities: The data itself (physical, digital, conceptual)
• Activities: Actions that generate or modify entities
• Agents: Persons, software, or organizations bearing responsibility

The most granular form of data lineage, tracing how individual columns in a target table are derived from specific source columns through transformation logic.

• Enables precise impact analysis for schema changes
• Essential for debugging complex ETL pipelines
• Tracks the exact SQL expression or function applied

The practice of storing unique snapshots of a dataset at specific points in time. This enables reproducibility of experiments, rollback to previous states, and direct comparison of data distributions.

• Often implemented via Delta Lake or Apache Iceberg
• Supports Time Travel queries for auditing
• Prevents training-serving skew in ML pipelines

A design pattern that identifies and tracks row-level changes (INSERT, UPDATE, DELETE) in source databases. It enables real-time synchronization of downstream systems without costly full reloads.

• Uses database transaction logs for minimal overhead
• Powers real-time event streaming architectures
• Foundational for maintaining an accurate Data Lineage record

A formal, machine-readable agreement between a data producer and its consumers. It explicitly defines the schema, semantics, and quality guarantees (SLOs) of the delivered data.

• Enforced programmatically at the pipeline level
• Prevents silent schema breakage
• Managed centrally in a Schema Registry

The lifecycle tracking of data's origins, movements, and transformations across systems. While data provenance focuses on the historical record of ownership and processing, data lineage maps the technical flow from source to consumption. It provides a complete audit trail, answering 'Where did this data come from and how was it altered?'

A World Wide Web Consortium standard defining a data model for representing and exchanging provenance information. It formalizes three core types: Entities (data), Activities (processes), and Agents (responsible parties). Using PROV ensures interoperability when sharing provenance records across different systems and organizations.

A chronological, tamper-proof record of all data access and modification events. This is the storage mechanism that makes provenance verifiable. Key properties include:

• Non-repudiation: Actions cannot be denied
• Append-only: Records cannot be altered retroactively
• Cryptographic chaining: Each entry is hashed and linked to the previous one

The practice of storing unique snapshots of datasets at specific points in time. Combined with provenance, versioning enables reproducibility—the ability to re-run an analysis on the exact same data state. Tools like Delta Lake and Apache Iceberg provide time travel capabilities that make this practical at scale.

The most granular form of lineage that traces how individual columns in a target table are derived from specific source columns through transformation logic. For example, tracking that customer_lifetime_value is calculated from orders.total and customers.acquisition_date. This granularity is essential for impact analysis and regulatory compliance.

An open standard and framework for collecting and propagating metadata about data lineage across job schedulers, query engines, and analytical tools. It provides a unified way to capture provenance from heterogeneous systems like Apache Spark, dbt, and Airflow, centralizing it into a single lineage graph for analysis.