Data Versioning

The foundational principle where each dataset change creates a new, immutable version, identified by a unique hash or tag. This prevents accidental overwrites and guarantees historical reproducibility.

• Key Mechanism: Content-addressable storage, where the identifier (e.g., a SHA-256 hash) is derived from the data's content.
• Example: Tools like DVC (Data Version Control) or LakeFS store pointers to committed data snapshots in Git, while the actual data resides in object storage (S3, GCS).
• Benefit: Any analysis can be precisely rerun by checking out the exact data version used originally.

The mechanism for recording the origin and transformation history of a dataset. It answers how a specific data version was created.

• Core Components: Captures upstream dependencies (source data versions), the transformation code (and its version), and execution parameters.
• Implementation: Often stored as metadata in a structured format (JSON, YAML) or within a dedicated ML Metadata Store.
• Critical for: Debugging data-related issues, compliance audits (GDPR, AI Act), and understanding the impact of upstream changes on model performance.

Optimization techniques to store only the changes between versions, rather than complete copies, to conserve storage space.

• Delta Encoding: Stores the difference (delta) between sequential versions. Common in columnar formats like Apache Parquet.
• Copy-on-Write vs. Merge-on-Read: Strategies for managing updates; Copy-on-Write (e.g., Apache Iceberg) creates new files for changes, favoring read speed. Merge-on-Read (e.g., Delta Lake, Hudi) writes changes to log files and merges them during read, favoring write speed.
• Impact: Enables versioning of massive datasets without prohibitive storage costs.

A query capability that allows users to query a dataset as it existed at a specific point in time or version tag.

• How it works: The versioning system maintains a transaction log (like Delta Log in Delta Lake) that maps timestamps or version IDs to specific data snapshots.
• Query Example: SELECT * FROM my_table VERSION AS OF '2024-01-15' or TIMESTAMP AS OF '2024-01-15 10:00:00'.
• Primary Use Cases: Reproducing past reports, debugging by comparing data states, and rolling back unintended changes without full restoration.

Applying Git-like branching workflows to data, allowing for isolated experimentation without affecting the primary (e.g., production) dataset.

• Process: A data engineer can create a branch from main, ingest new or transformed data, and test downstream pipelines, all in isolation.
• Merge & Conflict Resolution: Changes can be merged back, with systems providing conflict detection for schema or data overlaps.
• Tool Example: LakeFS provides full Git-like semantics (branch, commit, merge, rollback) for object-stored data lakes, enabling safe CI/CD for data.

The critical link where data versioning connects to model training and evaluation, ensuring full pipeline reproducibility.

• Orchestration: Tools like MLflow or Kubeflow can be configured to automatically record the version ID of training data as part of a model run.
• Traceability: This creates an immutable chain: Model Version v1.2 was trained on Data Version dataset@a1b2c3 using Code Commit abc123.
• Outcome: Complete audit trail for model governance and the ability to re-train models identically or diagnose performance drift by pinpointing data changes.

A software design pattern where the state of an application or dataset is derived from a sequence of immutable events stored as the single source of truth. This pattern is conceptually aligned with data versioning, as each event represents a state change.

• Core Principle: Instead of storing the current state, store all events that led to that state.
• Relation to Versioning: Replaying the event log up to a specific point reconstructs a historical version of the data.
• Benefit: Provides a complete audit log and enables temporal queries.

Data Lineage tracks the flow of data from its origin through various transformations and processes. Data Provenance refers to the detailed history of the data, including its origin and the sequence of processes applied.

• Relation to Versioning: Versioning provides the discrete points (snapshots) that lineage graphs connect. Each dataset version is a node, and transformations are edges.
• Critical for: Debugging, impact analysis (what models use this dataset version?), and regulatory compliance (GDPR, AI Act).
• Tools: OpenLineage, Apache Atlas, and data catalog integrations.