Active Metadata

Active metadata is not manually entered; it is automatically extracted from query logs, execution engines, and ETL pipelines in real-time. This creates a living map of the data estate.

• Push-based collection: Agents and crawlers emit metadata events as operations occur.
• Stream processing: Metadata is ingested via Kafka or Kinesis for immediate analysis.
• Example: A Spark job automatically publishes its input/output lineage to DataHub the moment it completes.

Unlike passive metadata used only for search, active metadata triggers automated workflows. The metadata layer becomes the control plane for the data platform.

• Policy enforcement: A schema change detected in metadata can automatically halt a downstream pipeline.
• Auto-scaling: Metadata about query load triggers dynamic cluster resizing.
• Example: An OpenLineage event showing a PII tag on a column automatically invokes a masking job.

Active metadata is stored in a property graph, not a flat table. This allows traversal of complex dependencies for impact analysis and root cause investigation.

• Node types: Tables, dashboards, ML models, pipelines, and users.
• Edge types: PRODUCES, CONSUMES, DERIVES_FROM, OWNS.
• Example: Tracing a failed dbt model node backward through the graph to find the exact upstream Airflow task that introduced a breaking schema change.

The system applies heuristics and ML models to the metadata stream to generate proactive suggestions, reducing manual toil for data engineers.

• Tiering suggestions: Identifies unused tables for cold storage based on query recency metadata.
• Join optimization: Recommends sort keys based on observed join patterns in query logs.
• Deprecation warnings: Alerts consumers when a source table has not been updated beyond its SLA.

Active metadata is exposed via robust, low-latency APIs to be consumed by other tools in the stack, not locked inside a single catalog UI.

• GraphQL/REST endpoints: Allow programmatic querying of lineage and ownership.
• Webhooks: Push notifications for specific metadata events (e.g., schema drift).
• Example: A CI/CD pipeline queries the metadata API to check if a proposed SQL migration violates existing data contracts before deployment.

Active metadata retains a full history of state changes, enabling point-in-time analysis and debugging of historical data quality issues.

• Auditability: Reconstruct the exact schema and ownership of a table from 30 days ago.
• Diffing: Compare the current metadata state against a previous snapshot to identify what changed.
• Example: A data scientist uses time travel to query the feature definitions that were active when a specific model version was trained, ensuring reproducibility.