Data Provenance

The foundational layer of provenance is a cryptographically verifiable record of a dataset's birth. This includes the source system, timestamp of creation, and the identity of the creator or responsible entity. By hashing the initial data and storing it on an immutable ledger, organizations create a tamper-proof birth certificate. This prevents retroactive manipulation and establishes a root of trust for all downstream lineage.

• Key Mechanism: Cryptographic hashing (SHA-256) at ingestion.
• Benefit: Non-repudiation of data origin.
• Example: A sensor recording its first reading with a signed hash.

Provenance must capture every state mutation a dataset undergoes. This is not just version control; it is a directed acyclic graph (DAG) of operations. Each node in the graph represents a specific transformation—such as a filtering query, a join operation, or a normalization function—along with its input and output schemas. This granular tracking enables precise debugging and impact analysis when a data quality issue is detected downstream.

• Captures: Code version, query logic, and execution timestamp.
• Granularity: Row-level or column-level lineage.
• Tooling: Often integrated with orchestrators like Apache Airflow or Dagster.

Raw lineage graphs are insufficient without rich, contextual semantic metadata. Provenance records must bundle the 'why' and 'how' alongside the 'what.' This includes the data card detailing the collection motivation, the steward responsible for quality, and the intended use constraints. This layer transforms a technical log into a governance artifact, enabling compliance officers to verify that data usage aligns with its consented purpose under regulations like GDPR and the EU AI Act.

• Includes: Data cards, ethical approvals, and legal basis for processing.
• Standard: W3C PROV-O ontology for semantic interoperability.

To move beyond internal trust, provenance data must be externally verifiable. This involves wrapping provenance assertions into W3C Verifiable Credentials. A data provider can issue a cryptographically signed claim about a dataset's attributes, which a consumer can verify without contacting the issuer. This creates a decentralized trust framework where the authenticity of a dataset can be mathematically proven, not just assumed, across organizational boundaries.

• Technology: Decentralized Identifiers (DIDs) and JSON Web Signatures.
• Use Case: Verifying a third-party training dataset's licensing before model ingestion.

The final characteristic is the immutability of the provenance log itself. By anchoring the cryptographic hashes of provenance records into a distributed ledger or a secure, append-only log, the system guarantees non-repudiation. No single administrator can alter the history of a dataset's journey. This provides the definitive audit trail required for regulatory investigations, proving exactly what data was used to train a specific model version at a specific point in time.

• Storage: Tamper-proof append-only logs.
• Outcome: Irrefutable evidence for legal and compliance audits.

The lifecycle tracking of data as it flows through ingestion, transformation, and storage pipelines. While provenance focuses on origin and authenticity, lineage maps the complete journey—every ETL step, join operation, and aggregation. Lineage enables impact analysis: when a source table changes, teams can instantly identify all downstream models and dashboards affected. Modern tools like Marquez and OpenLineage capture this metadata automatically, creating directed acyclic graphs of data dependencies.

The process of identifying which specific source or subset of training data caused a model's particular output. Critical for copyright compliance and debugging unexpected behaviors. Techniques include:

• Influence functions: Mathematically trace a prediction back to training examples
• Data Shapley values: Quantify each data point's marginal contribution
• Nearest neighbor retrieval: Find training samples most similar to a query

Attribution is becoming legally essential as courts examine whether AI outputs derive from infringing source material.

The practice of tracking and managing changes to datasets over time, analogous to Git for code. Every modification—schema changes, row additions, label corrections—receives a unique commit hash, enabling:

• Reproducibility: Recreate the exact dataset used in any experiment
• Rollback: Revert to a known-good state when data corruption occurs
• Auditability: Prove which data version a model was trained on

Tools like DVC and LakeFS integrate directly with ML pipelines, treating data as a versioned artifact alongside model weights and code.

A formal, machine-readable agreement between a data producer and its consumers that defines the schema, semantics, and quality guarantees of the data being provided. Contracts enforce provenance at the interface level:

• Schema enforcement: Reject data that violates column types or constraints
• Semantic guarantees: Define what null values mean, what units apply
• SLOs: Specify freshness, completeness, and accuracy thresholds

When a producer changes a schema, the contract triggers alerts to all downstream consumers before pipelines break. This shifts data governance left into the development cycle.

An adversarial attack where malicious data is injected into a training set to corrupt the model's learning process. Poisoning directly attacks provenance integrity by introducing inauthentic data into trusted pipelines. Attack types include:

• Backdoor attacks: Insert trigger patterns that cause specific misclassifications
• Label flipping: Corrupt ground truth labels to degrade accuracy
• Availability attacks: Maximize model error across all inputs

Defenses require robust provenance verification—cryptographic signing of data sources, anomaly detection on incoming batches, and continuous monitoring for distribution shifts.