Feature Lineage

Feature lineage captures the directed acyclic graph (DAG) of a feature's entire lifecycle. It tracks the specific source tables, streaming topics, or raw files from which data originated, and logs every intermediate transformation applied.

• Links raw source data to the final feature vector served in production.
• Records the exact SQL query, Python function, or Spark job that generated the value.
• Essential for answering the question: 'Where did this number come from?' during a model audit.

When a source schema changes or a data pipeline breaks, lineage graphs allow engineers to instantly identify every downstream feature, training dataset, and production model that will be affected.

• Prevents silent model degradation by proactively alerting teams to upstream breakages.
• Visualizes the blast radius of a data outage before it impacts customer-facing predictions.
• Enables safe deprecation of legacy features by confirming zero downstream consumers.

Lineage metadata is critical for reconstructing the exact state of the world at a historical timestamp. It ensures that training datasets are built with feature values as they existed, not as they are today.

• Stores the logic version (e.g., transformation_v2.py) active at the time of computation.
• Prevents data leakage by guaranteeing no future information contaminated the training set.
• Enables deterministic replay of historical features for backtesting and model validation.

For regulated industries like finance and healthcare, lineage provides the immutable audit trail required by model risk management (MRM) frameworks and regulations like SR 11-7 or the EU AI Act.

• Demonstrates exactly which raw data attributes influenced a credit decision or medical diagnosis.
• Provides cryptographic hashing or checksums of transformation code to prove immutability.
• Supports 'right to explanation' requests by mapping model inputs back to user-provided data.

When a model's prediction quality drops, lineage accelerates the troubleshooting process by isolating the problematic feature. Engineers can trace a suspicious value back through the pipeline to the exact point of failure.

• Differentiates between data drift (input distribution change) and pipeline bugs (code error).
• Compares the lineage of a failing prediction against a successful one to spot discrepancies.
• Reduces mean time to detection (MTTD) for data quality incidents from days to minutes.

Feature lineage is a core metadata component of a Feature Registry. Modern feature stores like Feast and Tecton automatically generate lineage graphs by parsing transformation definitions and source data connections.

• Lineage is automatically updated when a Feature View is modified or a new Feature Group is created.
• Ties directly into Data Observability platforms to correlate pipeline failures with feature staleness.
• Enables the 'Time Travel' capability by mapping timestamps to specific transformation logic versions.