Data Catalog

The engine that transforms a static data catalog into a dynamic operations hub. Unlike passive metadata that sits in a repository, active metadata is continuously harvested and used by automated systems to orchestrate data operations, enforce access policies, and recommend actions in real-time.

• Key difference: Passive metadata tells you what a table is; active metadata uses that information to auto-pause a pipeline when a schema changes.
• Feedback loop: Actions taken in downstream tools (dbt, Airflow) are fed back into the catalog, enriching context.
• Example: A catalog detects a column's null percentage spiking to 40% and automatically triggers a quality alert in the observability platform.

A formal, machine-readable agreement between a data producer and its consumers, enforced through the catalog. It defines the explicit schema, semantics, and quality guarantees (SLOs) of a data product.

• Catalog integration: Contracts are registered and versioned in the catalog, making guarantees discoverable before a consumer builds a pipeline.
• Breaking changes: If a producer attempts to deprecate a column, the contract validation engine checks downstream impact and blocks the change if it violates the agreement.
• Example: A customer_360 table contract guarantees a customer_id of type STRING with uniqueness > 99.9% and freshness < 1 hour.

The most granular form of data lineage, tracing how individual columns in a target table are derived from specific columns in source tables through transformation logic. This is the backbone of precise impact analysis.

• Beyond table lineage: Knowing a dashboard depends on a table is insufficient; column-level lineage pinpoints that the revenue metric comes from orders.total after a currency conversion.
• Parsing logic: Requires parsing SQL, Python, or Spark transformation code to build a fine-grained directed acyclic graph (DAG).
• Example: A catalog shows that marketing.attribution.revenue is derived from sales.orders.net_amount * finance.rates.usd_conversion.

An organization's ability to fully understand the health and state of data across its pipelines. It extends the catalog's role from inventory to continuous monitoring across five pillars: freshness, volume, schema, lineage, and quality.

• Catalog synergy: The catalog provides the baseline schema and lineage; observability tools detect anomalies against that baseline and push alerts back into the catalog.
• Incident context: When a pipeline breaks, the catalog surfaces the incident alongside the affected asset's ownership and lineage, accelerating root cause analysis.
• Example: An observability platform detects a schema change in a source table and updates the catalog's schema registry, triggering a notification to all downstream data product owners.

A decentralized sociotechnical architecture that organizes data ownership around business domains, treating data as a product. The data catalog becomes the federated marketplace where these products are discovered and consumed.

• Domain ownership: Each domain team owns its catalog entries, defining the metadata, contracts, and SLOs for its data products.
• Self-service: A global catalog stitches together domain-specific metadata, enabling any team to discover, understand, and request access to a data product without a central gatekeeper.
• Example: The 'Payments' domain publishes a transaction_facts product to the catalog with a defined schema, 99.95% uptime SLO, and a Slack channel for consumer support.

A centralized platform for defining, storing, and serving machine learning features consistently between training and inference. The data catalog serves as the discovery layer for these features, documenting their lineage back to raw source tables.

• Training-serving skew prevention: The catalog documents the exact transformation logic and source data for a feature, ensuring the same code is used in production inference.
• Point-in-time correctness: The catalog's lineage helps verify that feature values used for training accurately reflect the state of the world as it existed at the historical timestamp of the label.
• Example: A data scientist searches the catalog for customer_ltv_30d, finds it in the feature store, and sees its lineage traces back to transactions and customer_profiles tables with a 7-day freshness guarantee.