DataHub

DataHub's core innovation is its metadata change log, where all metadata events—like adding a dataset or updating a schema—are published as real-time streams (using Kafka or Pulsar). This enables:

• Real-time metadata propagation: Changes are reflected instantly across search, lineage, and governance tools.
• Event-driven integrations: External systems can subscribe to metadata events to trigger actions, like data quality checks or access policy updates.
• Decoupled, scalable components: The ingestion, serving, and indexing services operate independently, communicating via the metadata stream.

DataHub provides a Google-like search experience across all technical and business metadata. It uses Elasticsearch for indexing, enabling users to find data assets via:

• Keyword and semantic search: Search by table names, column descriptions, or tags.
• Faceted filtering: Narrow results by platform (Snowflake, BigQuery), owners, domains, or tags.
• Auto-complete and relevance ranking: Quickly find relevant datasets, dashboards, and pipelines.
• Integrated previews: View sample data, schema, and ownership directly in search results.

DataHub automatically captures and visualizes column-level lineage, showing how data flows from source to consumption. This includes:

• Upstream and downstream dependencies: See which pipelines, models, and reports depend on a specific column.
• Multi-platform support: Tracks lineage across databases (Snowflake, Redshift), transformation tools (dbt, Spark), and BI platforms (Looker, Tableau).
• Impact analysis: Understand the potential effects of schema changes or pipeline failures before they occur.
• Provenance tracking: Identify the origin and transformation logic applied to data.

Metadata is modeled using a schema-first approach with a typed graph model. The core concepts are:

• Entities: Represent core data assets like Datasets, Dashboards, Data Pipelines, and Users.
• Aspects: Collections of attributes describing an entity (e.g., a Dataset's schema, ownership, or usage statistics).
• Relationships: Define connections between entities (e.g., a Dataset Contains a Column, a User Owns a Dashboard). This flexible model allows for easy extension to support custom metadata types and new data tools.

Every action in the DataHub UI is backed by a GraphQL API and Python/Java SDKs, enabling full metadata automation. Use cases include:

• Automated metadata ingestion: Script the registration of new datasets or pipelines as part of CI/CD.
• Bulk metadata operations: Programmatically tag assets, assign owners, or apply governance policies.
• Custom integrations: Build internal tools that query lineage or embed DataHub's search.
• Metadata as Code: Define and version-control metadata definitions (like tags and domains) in git.

DataHub provides foundational tools for decentralized governance and team collaboration around data assets:

• Business Glossary & Domains: Organize assets into business domains and define standardized terms.
• Tagging & Classification: Apply structured tags (e.g., PII, Revenue) for search, filtering, and policy enforcement.
• Ownership & Documentation: Assign technical and business owners and enrich assets with Markdown documentation.
• Access Management: Integrate with SSO and group systems to control who can view or edit metadata.

A centralized inventory and management system that stores, organizes, and provides access to metadata about an organization's data assets. It is the foundational component for enabling data discovery, governance, and lineage tracking. Unlike a simple data dictionary, a full-featured catalog like DataHub provides:

• Search and discovery across tables, dashboards, and pipelines.
• Technical and business metadata unification.
• Automated metadata ingestion from various source systems.
• APIs for programmatic access and integration.

The tracking of data's origin, movement, transformations, and dependencies across systems and processes over its lifecycle. DataHub provides both table-level and column-level lineage, which is critical for:

• Impact analysis: Understanding what downstream reports or models will be affected by a schema change.
• Root cause debugging: Tracing an erroneous metric back to its source.
• Compliance and auditing: Demonstrating the provenance and handling of regulated data. Lineage is a core feature of metadata platforms, transforming a static catalog into a dynamic map of data flow.

A decentralized, domain-oriented sociotechnical framework for data architecture that treats data as a product. DataHub is a key enabling technology for implementing a Data Mesh by providing the necessary federated computational governance. It supports the mesh principles by:

• Allowing domain teams to own and document their data products.
• Providing a global search and discovery layer across all domains.
• Enforcing data contracts and quality standards through metadata.
• Facilitating interoperability between independently developed data products.

A service that manages and stores schemas (e.g., Avro, Protobuf, JSON Schema) for data in motion, typically in streaming pipelines using Kafka. While a metadata catalog like DataHub stores a broader set of metadata, a schema registry focuses on:

• Schema versioning and evolution history.
• Compatibility checking (backward/forward) to prevent breaking changes.
• Providing a central reference for serializers and deserializers. DataHub can integrate with schema registries to pull technical schema metadata, providing a unified view of both data-at-rest and data-in-motion.

A formal agreement between data producers and consumers that specifies the schema, semantics, quality, and service-level expectations (e.g., freshness, latency) for a data product. DataHub acts as the system of record and enforcement point for data contracts by:

• Storing the contract's schema and SLA definitions as metadata.
• Providing change notifications when schemas evolve.
• Enabling impact analysis to see which consumers rely on a given contract.
• Integrating with quality tools to monitor if SLAs are being met, turning static contracts into observable agreements.