Lineage Tracking (Data Provenance)

Nodes represent the core, versioned entities in the system. They are the vertices in the graph where edges connect.

• Data Nodes: Represent datasets, tables, or individual files. Each node is immutable and identified by a unique hash (e.g., SHA-256) of its content.
• Code Nodes: Represent scripts, notebooks, or pipeline definitions, also versioned via Git commit hashes.
• Model Nodes: Represent trained model artifacts (e.g., .pkl, .pt files) with associated metadata like architecture and framework.
• Run Nodes: Represent an execution instance that produced an output, linking to the specific code, input data, and parameters used.

Edges are the directed connections between nodes that explicitly define provenance and causality. They answer "what produced what?"

• Dataflow Edges: Show how data is transformed (e.g., raw_dataset --[cleaned_by]--> clean_dataset).
• Process Edges: Connect a Run Node to its input and output entities (e.g., training_run --[produced]--> model_v1).
• Derived From Edges: Indicate direct lineage (e.g., model_v2 --[derived_from]--> model_v1).
• Parametric Dependencies: Link a Run Node to the hyperparameter configuration file that governed its execution.

This is the descriptive information attached to nodes and edges, providing the context necessary for reproducibility and audit.

• Temporal Metadata: Timestamps for creation and modification.
• Provenance Metadata: User, execution environment (Docker image, library versions), and hardware specs (GPU type).
• Operational Metadata: System metrics like runtime, compute cost, and status (success/failure).
• Custom Tags: Key-value pairs for business context (e.g., project: fraud_detection, regulatory: gdpr).

The persistent, versioned storage backend that guarantees the lineage graph's nodes are tamper-proof and permanently accessible.

• Content-Addressable Storage (CAS): Artifacts (data, models) are stored under a key derived from their cryptographic hash. Any change creates a new, unique node.
• Examples: Object stores (S3, GCS) with immutable versioning enabled, or specialized systems like DVC-managed storage.
• Integrity: The hash in the graph node must always resolve to the exact artifact bytes, enabling verification of the entire lineage chain.

The ability to traverse the graph forward from a given node to identify all dependent entities. This is critical for assessing the blast radius of changes or defects.

• Use Case: Identifying all models trained on a dataset found to have a quality issue.
• Process: Starting from a data node, follow all outgoing produced or derived_from edges to find affected models, reports, and deployments.
• Output: A complete list of assets that may be invalidated and require retraining or review.

The ability to traverse the graph backward from a given node to discover its complete origin. This is essential for debugging and compliance.

• Use Case: Explaining why a model's performance degraded. Trace back through training runs, data preprocessing steps, and source data versions.
• Process: Starting from a model node, recursively follow incoming derived_from and used edges to reconstruct its exact generation history.
• Output: A deterministic causal chain pinpointing the specific code commit, parameter change, or data shift responsible for an observed outcome.

When a model's performance degrades or an unexpected prediction occurs, lineage tracking enables precise root cause analysis. By tracing the exact data inputs, code version, and hyperparameters used for a specific training run, engineers can recreate the exact environment to debug issues. This is critical for diagnosing failures stemming from data drift, code regressions, or contaminated training sets. Without lineage, debugging becomes a process of guesswork.

In regulated industries (finance, healthcare, pharmaceuticals), organizations must demonstrate the provenance of data and models used in automated decision-making. Lineage provides the immutable audit trail required for frameworks like GDPR, HIPAA, and the EU AI Act. Auditors can verify:

• The origin of training data and its compliance with usage rights.
• That bias mitigation steps were applied to specific datasets.
• The complete chain of custody for a model deployed in a clinical or financial context.

Lineage maps downstream dependencies, answering the critical question: "If this dataset or feature changes, which models and business reports will be affected?" This allows for proactive governance. For example:

• Before retiring an old data pipeline, engineers can identify all models that depend on its output.
• If a data quality issue is detected in a source table, teams can immediately assess the blast radius and prioritize model re-training.
• This transforms data management from reactive to strategic.

Implementing ethical AI principles requires verifiable proof of practices. Lineage tracking operationalizes governance by logging:

• Which bias detection or fairness metrics were calculated during training and on what data slices.
• The version of the debiasing algorithm or synthetic data generator used.
• The provenance of data subject to differential privacy or other privacy-enhancing technologies. This creates a defensible record that the model was developed with due diligence, supporting algorithmic impact assessments.

Lineage is integral to data observability. By instrumenting pipelines to track lineage, teams can monitor the health and freshness of data flowing into models. Combined with data quality checks, lineage can trigger alerts when:

• A critical upstream data source fails, breaking the lineage graph.
• Stale data is detected in a feature store used for online inference.
• Anomalous statistical properties (data drift) are detected in a lineage-linked feature. This ensures models are making predictions on reliable, current data.

In large ML teams, lineage serves as a system of record that transcends individual knowledge. It answers questions like:

• "Who trained this model and what was their rationale for this parameter set?"
• "Has anyone already built a feature similar to the one I need?"
• "Which experiment run produced the best model for a similar task last quarter?" By making lineage discoverable, it reduces duplicate work, accelerates onboarding, and ensures institutional knowledge is preserved when team members change roles.

Artifact storage is the system for versioning and persisting large, immutable outputs from machine learning runs. It is a critical dependency for lineage tracking.

• Key Artifacts: Trained model files (e.g., .pt, .h5), datasets, visualizations, and serialized preprocessing objects (e.g., LabelEncoder instances).
• Lineage Link: Each artifact is tagged with the Run ID that produced it, creating a direct, queryable link back to the exact code, data, and parameters used in its creation.
• Storage Backends: Typically uses remote object stores like Amazon S3, Google Cloud Storage, or Azure Blob Storage, integrated with metadata catalogs.

A pipeline run is a single execution instance of a multi-step machine learning workflow (e.g., data prep → training → evaluation). Lineage tracking operates at this granular level.

• Step-Level Provenance: Each step's inputs (data, parameters), outputs (artifacts), code version, and environment are logged.
• DAG Tracking: The directed acyclic graph (DAG) of step dependencies is recorded, showing how data and models flow through the pipeline.
• Use Case: Essential for debugging failures, understanding the impact of a data change on a final model, and meeting regulatory audit requirements where every transformation must be documented.

An environment snapshot is a complete, versioned record of the software context used during a machine learning run. It is a non-negotiable element of reproducible lineage.

• Contents: Includes all library versions (from pip freeze or conda env export), system packages, environment variables, and hardware details (e.g., GPU driver version).
• Provenance Role: Without this snapshot, the lineage of a model is incomplete; the same code and data can produce different results with different library versions.
• Implementation: Often captured as a requirements.txt file, a Conda environment.yml, or a container image hash (e.g., Docker SHA).

Run metadata encompasses all ancillary information logged alongside a machine learning experiment. It provides the contextual "who, when, and why" for lineage records.

• Core Fields: Includes the initiating user, start/end timestamps, Git commit hash, branch name, and the Run ID.
• Custom Tags: Teams add key-value pairs like project=churn_prediction, hypothesis=test_new_encoder, or regulatory_audit=true to filter and organize lineage queries.
• Audit Trail: This metadata creates an immutable audit trail, crucial for compliance (e.g., GDPR, EU AI Act) and internal governance, answering questions about model authorship and change justification.

A model registry is a centralized system for managing the lifecycle of trained models. It extends lineage tracking from experimentation into production deployment and governance.

• Versioned Lineage: Each model version in the registry is linked to its originating experiment run, training data snapshot, and evaluation metrics.
• Stage Tracking: Manages model stages (e.g., Staging, Production, Archived) and records promotion/demotion events, who approved them, and associated documentation.
• Deployment Provenance: When a model is deployed to a production endpoint, the registry provides the full lineage needed for incident response, rollback decisions, and compliance reporting.