DataHub

Unlike traditional pull-based catalogs, DataHub ingests metadata via a stream-first architecture using Kafka and Metadata Change Events (MCEs). This enables near-real-time propagation of changes across the platform. Key characteristics include:

• Log-based ingestion: All metadata changes are immutable events, enabling replay and audit.
• Asynchronous processing: Metadata ingestion, profiling, and indexing are decoupled for resilience.
• Push model: Producers emit changes directly, eliminating the latency of scheduled crawls.

DataHub models metadata as an interconnected graph of entities (datasets, dashboards, pipelines, ML models) and relationships (lineage, ownership, tags). This graph-native approach allows:

• Traversal queries: Navigate upstream lineage from a dashboard to its source tables.
• Impact analysis: Identify all downstream consumers of a deprecated column.
• Flexible extensions: Define custom entity types and relationship edges without schema migrations.

DataHub supports a plugin-based ingestion framework that connects to diverse data sources. Metadata Ingestion Sources extract metadata and emit MCEs. Supported sources include:

• Data warehouses: Snowflake, BigQuery, Redshift
• Streaming platforms: Kafka, Pulsar
• Orchestrators: Airflow, Dagster
• BI tools: Looker, Tableau, Power BI
• ML platforms: MLflow, SageMaker Each source can be configured with its own schedule and filtering rules.

Metadata is indexed into Elasticsearch to power full-text search, faceted browsing, and ranked relevance. Users can discover assets using:

• Natural language queries: 'customer churn model training data'
• Faceted filters: Platform, domain, owner, tags, glossary terms
• Ranking signals: Usage frequency, freshness, certification status This transforms the catalog from a passive registry into an active discovery engine.

DataHub operationalizes governance through metadata tests and assertions that run continuously. Capabilities include:

• Schema compliance checks: Verify that production schemas match registered contracts.
• Classification auto-tagging: Automatically apply glossary terms and sensitivity labels based on column names or data patterns.
• Freshness monitoring: Alert when a table's last update exceeds its SLA.
• Ownership propagation: Inherit ownership from parent domains to child assets.

DataHub renders column-level lineage as an interactive directed acyclic graph (DAG). Users can:

• Trace upstream: Identify all transformations contributing to a metric.
• Trace downstream: Assess the blast radius of a schema change.
• Filter by time: View lineage as it existed at a specific point in history.
• Export lineage: Integrate with external governance tools via the OpenLineage standard. The visualization supports both coarse-grained (table) and fine-grained (column) views.

Metadata that is continuously harvested and used by automated systems to orchestrate data operations, enforce policies, and recommend actions in real-time. DataHub is built on this principle, using a stream-oriented infrastructure where metadata changes trigger immediate actions.

• Key Mechanism: Metadata Event Streams (MES) power real-time subscriptions
• Contrast: Passive metadata sits in a static catalog; active metadata drives automation
• Example: A schema change detected by DataHub can instantly notify downstream owners and update the data contract

A formal, machine-readable agreement between a data producer and its consumers that defines the schema, semantics, and quality guarantees of the delivered data. DataHub enables contract enforcement through its assertions and schema verification capabilities.

• Components: Schema definition, SLAs, semantic meaning, and ownership
• Enforcement: Automated checks on freshness, volume, and schema compatibility
• Integration: Works with Schema Registry and dbt to validate contracts at build time

A decentralized sociotechnical architecture that organizes data ownership around business domains, treating data as a product. DataHub serves as the federated governance layer in a mesh, enabling domain autonomy while maintaining global discovery standards.

• Domain Modeling: DataHub's Domains feature maps directly to mesh business units
• Data Products: Assets can be grouped and documented as discoverable products
• Federated Governance: Central policies with decentralized ownership and curation