Data Domain Inference

The core function is to assign a semantic label to a column based on its content. This is distinct from a data type (e.g., string). For example, a string column containing 'USD', 'EUR', 'JPY' would be inferred as the domain currency_code, not just text. Common domains include:

• Geographic: country, city, postal_code, latitude/longitude
• Temporal: date, timestamp, year, month_name
• Financial: currency, stock_ticker, transaction_id
• Personal: email_address, phone_number, person_name
• Product: sku, product_category, manufacturer

Inference engines analyze the syntactic patterns and value formats within a column. They use regular expressions, lookup tables, and statistical analysis to match values against known domain patterns.

Examples:

• A column where 95% of values match the regex ^[A-Z]{3}$ and correspond to entries in the ISO 4217 list is inferred as currency_code.
• Values following patterns like (XXX) XXX-XXXX or international formats trigger a phone_number inference.
• Columns with values like [email protected] are classified as email_address. The system evaluates the confidence of the match based on the proportion of conforming values.

Algorithms examine cardinality (number of distinct values), value distribution, and statistical properties to support domain inference.

How it works:

• A column with very high cardinality (near-unique values) might be an id or transaction_hash.
• A column with low cardinality and a specific set of values (e.g., 'Active', 'Inactive', 'Pending') is likely a status domain.
• For numeric columns, the range and distribution help: a column of integers between 1 and 120 might be age, while floats between 0.0 and 1.0 could be probability_score. This analysis works in conjunction with pattern matching to disambiguate similar formats.

Advanced inference uses the context provided by other columns in the same table or dataset. The relationship between columns can clarify ambiguous domains.

Example Scenarios:

• A column named code with values 'US', 'UK', 'DE' might be ambiguous. If another column is inferred as country_name, then code can be confidently inferred as country_code.
• A numeric column could be price. If a neighboring column is inferred as currency_code, this reinforces the price domain inference. This mimics how a human analyst uses surrounding information to understand a column's purpose.

Inferred domains are critical metadata fed into data catalogs and governance platforms. They automate the population of business glossaries and enable:

• Automated Data Classification: Tagging columns containing email_address or social_security_number as sensitive PII for compliance (e.g., GDPR, CCPA).
• Improved Discoverability: Analysts can search for all tables containing customer_id or product_sku across the organization.
• Schema Standardization: Identifying columns that serve the same semantic purpose but have different names (e.g., cust_id, client_id, customer_number) for consolidation efforts.

Beyond rule-based systems, supervised and unsupervised machine learning models can be trained to infer domains, especially for custom or complex business-specific categories.

Approaches include:

• Supervised Classification: Training a model (e.g., Random Forest, Transformer) on labeled examples of columns and their known domains (e.g., invoice_number, clinical_trial_id).
• Embedding-Based Similarity: Generating embeddings for column values and names, then comparing them to a library of known domain embeddings to find the closest match.
• Ensemble Methods: Combining the confidence scores from rule-based pattern matching, statistical analysis, and ML models to make a final, high-confidence domain prediction. This is essential for handling noisy, real-world enterprise data.

The automated process of determining the logical storage type of a column's values, such as integer, string, datetime, or boolean. This is a foundational step that precedes semantic domain inference.

• Examples: Identifying a column of numbers as int64 or a column of text as varchar(255).
• Contrast with Domain Inference: Type inference deals with format and storage; domain inference deals with semantic meaning (e.g., a string column could be a 'city name' or a 'product SKU').

The automated identification of recurring formats or structures within data values. This technique directly fuels domain inference by detecting tell-tale signatures.

• Key Patterns for Domains:
  • \d{3}-\d{2}-\d{4} for U.S. Social Security Numbers.
  • [A-Z]{3} for currency codes (ISO 4217).
  • ^[\w\.-]+@[\w\.-]+\.\w+$ for email addresses.
• Implementation: Often uses regular expressions, statistical frequency analysis, or learned classifiers.

A specialized form of domain inference focused on identifying columns containing Personally Identifiable Information for compliance (GDPR, CCPA) and security.

• Common PII Domains: email_address, phone_number, credit_card, national_id, ip_address.
• Methods: Combines pattern matching, lookup against known formats (e.g., Luhn algorithm for credit cards), and context analysis (column names like 'user_ssn').
• Critical Output: Triggers data masking, encryption, or access control policies.

The automated process of inferring the structural metadata of a dataset, including column names, inferred data types, constraints (nullable/unique), and high-level relationships. Domain inference provides the semantic layer atop this discovered schema.

• Process Flow: Raw Data → Schema Discovery (Structure) → Data Profiling (Statistics) → Domain Inference (Semantics).
• Output: A proposed schema definition (e.g., SQL CREATE TABLE statement, Avro schema) that includes discovered types and potential keys.

The process of identifying and linking records that refer to the same real-world entity across disparate sources. Accurate domain inference is crucial for determining which attributes are valid for matching.

• Dependency on Domain Inference: To match 'customer' records, the system must first correctly infer domains like person_name, postal_address, and email. Matching on an incorrectly inferred domain (e.g., treating a product code as a name) causes failures.
• Example: Recognizing that 'Jon Doe Corp.', 'J. Doe Corporation', and 'John Doe Corp.' refer to the same business entity.

The broader governance process of categorizing data assets based on sensitivity, business value, and regulatory requirements. Domain inference automates the initial, content-based tagging that feeds into classification policies.

• Hierarchy: Domain Inference (technical/semantic tag) → Business Glossary Mapping → Data Classification (policy label like 'Confidential', 'Public').
• Use Case: A column inferred as credit_card_number is automatically classified as 'Restricted - PII', applying encryption-at-rest and strict access controls.