Data Discovery

This capability involves the programmatic collection of metadata from diverse source systems—databases, data pipelines, BI tools, and cloud storage—into a central catalog. It is foundational for discovery, as an empty catalog is useless.

• Key Sources: Relational databases (via JDBC/ODBC), data warehouses (Snowflake, BigQuery), pipeline orchestrators (Airflow, dbt), and file systems (S3, HDFS).
• Methods: Connectors use APIs, query system tables, or parse logs to extract technical metadata (schemas, data types) and operational metadata (refresh times, owners).
• Challenge: Maintaining sync frequency to reflect real-time changes without overloading source systems.

This is the primary user interface for discovery, moving beyond simple keyword matching. It allows users to find data using natural language queries and business context.

• Semantic Understanding: Uses techniques like vector search on embedded column descriptions and table names to find conceptually similar assets, even without exact keyword matches.
• Faceted Browsing: Enables filtering by critical dimensions like data domain (e.g., 'finance', 'customer'), owner, freshness, classification tags (e.g., 'PII'), and popularity.
• Example: A search for 'monthly recurring revenue' returns the finance.mrr_summary table, the 'MRR' column in the subscriptions table, and the related 'Annual Recurring Revenue' dashboard.

This capability provides a statistical snapshot of a dataset without requiring users to query the underlying data directly. It builds immediate understanding and trust.

• Profile Metrics: Automatically generates summary statistics for each column: value distributions, min/max, mean, null counts, and cardinality.
• Data Preview: Shows a safe sample (e.g., first 100 rows) to confirm content and format.
• Quality Indicators: Highlights potential issues like high null percentages, skewed distributions, or unexpected data types, serving as an initial data quality check.

This maps the data journey, showing upstream sources and downstream dependencies of any asset. It is critical for understanding provenance and assessing change impact.

• Visual Graphs: Displays column-level lineage showing how a specific metric is calculated from raw source columns through various transformations.
• Use Cases: Root-cause analysis for a broken dashboard (trace back to the failed pipeline). Impact analysis before modifying a schema (see all dependent reports and models).
• Integration: Relies on parsing pipeline code (SQL, Spark) or integrating with orchestration tools to build the dependency graph.

This transforms a technical catalog into a living knowledge base by capturing human expertise and business context around data assets.

• Annotations: Allows users to add descriptions, usage notes, and rating systems.
• Business Glossary Links: Connects technical columns to defined business terms (e.g., links column cust_id to the business term 'Active Customer').
• Ownership & Stewardship: Assigns data owners (accountable) and data stewards (operational) to ensure someone is responsible for the asset's definition and quality.

This uses usage telemetry and collaborative filtering to proactively suggest relevant data assets to users, accelerating discovery.

• Popularity Signals: Tracks metrics like query frequency, user bookmarks, and downstream dependency count to surface 'most trusted' or 'most used' datasets.
• Personalized Recommendations: Suggests assets based on a user's search history, domain of work, or similarity to assets used by peers in their department.
• Example: An analyst viewing a customer table might be recommended the related 'customer_segments' model and the 'regional_sales' dashboard that frequently uses it.

A centralized inventory and management system that stores, organizes, and provides access to metadata about an organization's data assets. It is the foundational technology that powers data discovery by indexing technical, operational, and business metadata, enabling search, browsing, and lineage visualization. Key functions include:

• Automated metadata ingestion from databases, pipelines, and BI tools.
• Search and faceted filtering to locate datasets by name, tag, or column.
• Integrated data profiling and preview to assess fitness for use.
• Social features like ratings, user annotations, and stewardship assignments.

The automated analysis of a dataset to uncover its structure, content, quality, and relationships. It is a critical input to the discovery process, providing statistical summaries that help users understand what data contains before they decide to use it. Profiling typically examines:

• Column-level statistics: Data types, uniqueness, null counts, value distributions, and patterns.
• Data quality indicators: Identification of outliers, format violations, and referential integrity issues.
• Inter-column relationships: Discovery of potential primary/foreign keys and correlations.

Tools like Great Expectations and Soda Core automate this process, generating profiles that populate catalog entries.

A curated collection of definitions for key business terms, concepts, and metrics. It establishes a common organizational vocabulary, ensuring consistent interpretation of data. During discovery, glossary terms are linked to physical data assets (tables, columns, reports), allowing users to search for "Customer Lifetime Value" and find all related datasets and calculations. This bridges the gap between technical implementation and business meaning, a core goal of effective data discovery.

The tracking of data's origin, movement, transformations, and dependencies across systems over its lifecycle. Forward lineage shows where data flows to; backward lineage shows where it came from. In discovery, lineage provides critical context:

• Impact Analysis: Understand what reports or models will be affected if a source table changes.
• Root Cause Debugging: Trace an erroneous metric in a dashboard back to its source.
• Trust and Compliance: Verify data provenance for regulatory audits.

Column-level lineage offers the most granular view, mapping transformations at the individual field level.

A reusable, high-quality data asset—such as a curated dataset, machine learning model, or API—that is packaged, documented, and managed as a product. Data products are the primary discoverable units in a Data Mesh architecture. Each product has a clear owner, SLOs (e.g., for freshness and quality), and a data contract defining its schema and semantics. Discovery platforms surface these products, enabling consumers to find reliable, well-governed data with known service levels, moving beyond discovering raw tables.

A business representation of data that sits between physical data stores and consuming applications (e.g., BI tools). It translates complex data structures into familiar business terms, metrics, and logic (e.g., 'Revenue' defined as Sum(Sales_Amount) - Sum(Discount)). This layer powers semantic discovery, allowing users to search for business concepts rather than table names. Tools like Cube, AtScale, and LookML implement semantic layers, which are often integrated with catalogs to provide a unified discovery experience for both technical and business users.