Data Lineage

Traces data from its final output back to its raw source systems. This is critical for regulatory audits and debugging.

• Answers: 'Where did this data come from?'
• Maps transformations, joins, and aggregations in reverse
• Essential for verifying that model training data originated from a lawful, specified purpose

Tracks data from its source to all downstream consumers and applications. This enables impact analysis before making pipeline changes.

• Answers: 'What systems will break if I modify this schema?'
• Identifies all models, dashboards, and APIs dependent on a specific data field
• Prevents unintended repurposing by exposing all usage points

Captures transformations at the individual attribute level, not just the table or dataset level. This is mandatory for purpose limitation controls.

• Tracks how a specific PII field was masked, tokenized, or filtered
• Validates that derived features remain within the bounds of original consent
• Provides forensic evidence for data subject access requests

Uses SQL parsers, query log analysis, and API instrumentation to build lineage maps without manual tagging. Accuracy depends on coverage.

• Extracts logic from ETL scripts, dbt models, and Spark jobs
• Handles dynamic SQL and procedural code through runtime instrumentation
• Reduces human error in compliance documentation

Maintains a historical record of how data definitions and pipelines evolved over time. A point-in-time audit requires knowing the exact logic that produced a past result.

• Snapshots lineage graphs alongside data versions
• Enables auditors to replay historical transformations
• Critical for defending against claims of retrospective data misuse

Maps data movement across heterogeneous environments: from on-premise databases to cloud warehouses, feature stores, and model training pipelines. Boundary crossing is where repurposing risk is highest.

• Tracks data as it leaves the system of record
• Integrates with data catalogs and policy enforcement points
• Validates that data entering a training set has not violated its original use limitation

A chronological, immutable record of all data access, modification, and usage events. While lineage maps the flow, the audit trail provides the forensic evidence of who touched what data and when.

• Verifies processing remained within specified purposes
• Uses WORM storage (Write Once, Read Many) for non-repudiation
• Essential for demonstrating compliance during regulatory investigations

The architectural component that intercepts data access requests and enforces authorization decisions. Lineage metadata informs the PEP about data's provenance and permitted uses.

• Evaluates context: user, resource, environment, and action
• Dynamically blocks repurposing attempts in real-time
• Integrates with Attribute-Based Access Control (ABAC) engines

The practice of logically or physically segregating datasets to prevent data collected for one model from being reused by another. Lineage graphs validate that isolation boundaries remain intact.

• Uses namespace partitioning in data lakes
• Prevents cross-contamination between business units
• Enforced through information barriers (ethical walls)

The legal and technical requirement to explicitly define processing objectives before data collection. Lineage provides the runtime verification that actual data movement matches the declared purpose.

• Documented in machine-readable policy-as-code
• Prevents function creep in ML workflows
• Required under GDPR Article 5(1)(b)

The automated monitoring of data pipelines to detect anomalies and lineage breaks before they degrade downstream model performance. Lineage is the map; observability is the radar.

• Tracks freshness, volume, schema, and distribution drift
• Alerts on unauthorized data joins or transformations
• Ensures continuous compliance with quality SLAs

A secure deletion method that renders data permanently inaccessible by destroying the encryption keys. Lineage graphs identify all downstream copies and derivatives that must also be crypto-shredded.

• Enables verifiable right to erasure fulfillment
• More reliable than overwriting distributed storage
• Critical for enforcing data retention expiry across complex pipelines