Data Catalog

This is the ingestion engine of a data catalog. It uses connectors and crawlers to automatically scan and inventory data assets across disparate sources—databases (PostgreSQL, Snowflake), data lakes (S3, ADLS), BI tools (Tableau, Power BI), and pipelines (Airflow, dbt).

• Technical Harvesting: Extracts technical metadata like schemas, table/column names, data types, and partition structures.
• Operational Metadata: Captures lineage (upstream sources, downstream consumers), refresh frequency, and data freshness metrics.
• Process: Often uses change data capture (CDC) or scheduled scans to maintain an up-to-date inventory without manual intervention.

The primary user interface for data consumers. It enables Google-like search across all indexed metadata, moving beyond simple keyword matching.

• Semantic & Faceted Search: Users can search by business terms (e.g., "customer lifetime value") and filter by facets like data domain, owner, freshness, or PII classification.
• Business Glossary Integration: Maps technical column names (e.g., cust_acct_num) to approved business terms ("Customer Account ID"), bridging the IT-business gap.
• Popularity & Usage Signals: Ranks search results based on usage statistics, query frequency, and user ratings to surface the most trusted and relevant datasets first.

Provides visual, traceable maps of data movement and transformation. This is critical for root-cause analysis, regulatory compliance, and understanding data dependencies.

• End-to-End Lineage: Traces a column in a dashboard report back through ETL/ELT transformations to its raw source system.
• Impact Analysis: Predicts which downstream reports, models, or APIs will be affected if a source schema changes or data quality breaks.
• Provenance: For Retrieval-Augmented Generation (RAG), lineage proves the origin of data used to ground an AI response, which is essential for hallucination mitigation and auditability.

The control plane for enforcing data policies, security, and quality standards. It operationalizes governance by attaching rules directly to assets.

• Sensitive Data Classification: Automatically tags columns containing PII, PHI, or financial data using pattern matching or ML classifiers.
• Access Control & Masking: Integrates with IAM systems to enforce column- and row-level security policies; can suggest dynamic data masking rules.
• Stewardship Workflows: Assigns data owners and stewards, and manages workflows for certifying datasets, approving glossary terms, and resolving quality issues.

Turns the catalog into a collaborative platform, building collective intelligence around data assets and reducing tribal knowledge.

• Annotations & Ratings: Users can add descriptions, warnings, or rate datasets for quality, similar to product reviews.
• Usage Documentation: Allows consumers to document common queries, known issues, and example use cases directly on the asset page.
• Subscription & Notifications: Users can subscribe to datasets to be alerted of schema changes, quality incidents, or certification status updates.

The catalog acts as a central metadata hub, providing APIs and integrations to activate metadata across the modern data stack.

• ML Feature Store Integration: Catalogs features, their definitions, and statistical profiles for model reproducibility.
• RAG System Integration: Serves as the authoritative source for document metadata (owner, freshness, domain) in hybrid retrieval systems, enabling smarter document chunking and source attribution.
• Pipeline Orchestration: Feeds certified data asset lists and quality scores into tools like Apache Airflow to trigger or halt downstream pipelines.
• API-First Design: Offers RESTful APIs for automated metadata lookup, enabling data-driven applications to programmatically verify data sources.

Data Lineage is the detailed tracking and visualization of data's lifecycle, documenting its origins, movements, transformations, and dependencies across systems. While a Data Catalog inventories assets and their metadata, lineage provides the dynamic map of how data flows and changes.

• Purpose: Critical for impact analysis (e.g., understanding what breaks if a source column changes), debugging data pipelines, and proving compliance with regulations like GDPR or CCPA.
• Relation to Catalog: A mature Data Catalog often integrates lineage information, showing upstream sources and downstream consumers for each dataset, turning a static inventory into a living map of data dependencies.

Data Governance is the overarching framework of policies, standards, roles, and processes that ensure the formal management of data assets across an organization. A Data Catalog is the primary technical enabler of a governance program.

• Key Functions it Supports:
  • Stewardship: Assigning data owners and stewards for accountability.
  • Quality Management: Tagging datasets with quality scores and rules.
  • Security & Compliance: Tagging data with classification levels (PII, Confidential) and masking policies.
  • Access Control: Managing who can discover and request access to data.
• Without a Catalog, governance policies are difficult to implement and enforce, as there is no system of record for data assets.

Metadata Management is the discipline of handling the information that describes other data. A Data Catalog is an active metadata management platform—it doesn't just store metadata; it makes it actionable.

• Types of Metadata Managed:
  • Technical: Schema, data types, lineage, storage location.
  • Business: Descriptions, glossary terms, ownership, usage ratings.
  • Operational: Freshness, popularity, query frequency, pipeline run status.
• Modern catalogs use active metadata and machine learning to auto-tag data, infer relationships, and recommend assets, moving beyond passive repositories.

Data Mesh is a decentralized sociotechnical paradigm for enterprise data architecture, organizing data by business domains (e.g., marketing, finance) with domain teams owning their data as products. A Data Catalog is the critical interoperability layer in a Data Mesh.

• Role of the Catalog: It provides the global marketplace where domain teams publish their data products with standardized contracts (schema, SLA, quality metrics).
• Enables Discovery: Consumers from other domains can search, understand, and access these federated data products without central team bottlenecks. The catalog enforces the mesh's principles of discoverability and self-service.

A Data Marketplace (or Data Portal) is a consumer-facing interface, often powered by a Data Catalog, that allows users to browse, search, preview, and request access to curated data assets within an organization. It emphasizes the consumption experience.

• Key Features:
  • Curated Collections: Like an app store, featuring high-quality, certified datasets.
  • Self-Service Provisioning: Integrated access requests and automated approvals.
  • User Ratings & Reviews: Community feedback on data usability and quality.
• Relation to Catalog: The marketplace is the storefront; the catalog is the backend inventory, governance, and management system that supplies it.

Master Data Management (MDM) is the process of creating and maintaining a single, consistent, and authoritative source of truth for critical business entities (e.g., Customer, Product, Supplier). A Data Catalog and MDM are complementary, not competing.

• How They Work Together:
  • MDM creates the golden record—the definitive version of a customer's data, mastered from multiple systems.
  • Data Catalog discovers and inventories all source systems that feed into MDM, documents the mastering rules and workflows, and then publishes the golden record as a certified data product for enterprise-wide discovery and use.
• The catalog makes the output of MDM discoverable and governable across the organization.