Data Cleansing

Deduplication is the process of identifying and merging or removing duplicate records that refer to the same real-world entity. It is critical for preventing data inflation and ensuring a single source of truth within a knowledge graph.

• Key Methods: Rule-based matching (exact keys), probabilistic matching, and fuzzy matching using algorithms like Levenshtein distance or Jaro-Winkler similarity.
• Example: Identifying that "J. Smith, 123 Main St" and "John Smith, 123 Main Street" are the same person.
• Impact: Eliminates redundant nodes, reduces storage costs, and prevents conflicting facts.

Standardization converts data into a consistent format, while canonicalization reduces multiple valid representations to a single, authoritative form. This creates uniformity essential for semantic mapping and querying.

• Common Targets: Dates (→ YYYY-MM-DD), phone numbers (→ E.164 format), addresses (→ parsed street, city, ZIP), and units of measurement.
• Process: Applies formatting rules, lookup tables, and regular expressions.
• Purpose: Enables accurate entity linking, schema alignment, and reliable aggregation by ensuring all instances of a value are identical.

This technique enforces data integrity by checking values against predefined rules, formats, ranges, or referential constraints. It identifies records that violate the expected data model.

• Types of Rules:
  • Syntactic: Correct data type (e.g., integer, email format).
  • Semantic: Logical validity (e.g., end_date must be after start_date).
  • Domain: Adherence to business rules (e.g., product SKU must exist in master catalog).
• Outcome: Records are flagged for correction, rejected, or handled by exception workflows.

Imputation is the process of replacing missing, null, or blank values with substituted estimates. The method chosen significantly affects statistical analysis and machine learning model training.

• Simple Techniques: Using a statistical measure (mean, median, mode) or a constant (e.g., "Unknown").
• Advanced Techniques: Model-based imputation (k-Nearest Neighbors, regression) or leveraging relationships within the knowledge graph to infer missing attributes.
• Consideration: The choice between imputation and leaving data missing depends on the downstream use case and the Missing Completely at Random (MCAR) assumption.

Outliers are data points that deviate significantly from the majority of the dataset. Detection identifies them, and treatment decides how to handle them to prevent skewing analysis.

• Detection Methods: Statistical (Z-score, IQR), proximity-based, or clustering.
• Treatment Strategies:
  • Capping/Winsorizing: Replacing extreme values with a specified percentile.
  • Transformation: Applying mathematical functions to reduce skew.
  • Investigation: Outliers may be valid but rare events (e.g., fraud), requiring domain expertise.
• Goal: Improve model robustness and the accuracy of descriptive statistics.

Parsing deconstructs complex, semi-structured fields into discrete, atomic components. Structural normalization reorganizes nested or hierarchical data into a flat, tabular structure suitable for integration.

• Common Use Cases:
  • Splitting a full name field into first_name, last_name.
  • Flattening JSON or XML payloads into relational tables or RDF triples.
  • Extracting key-value pairs from log files or free-text comments.
• Tooling: Often implemented using custom scripts, ETL tools, or declarative mapping languages like RML (RDF Mapping Language) for knowledge graph population.

The process of standardizing data from disparate sources by resolving syntactic, structural, and semantic differences to create a unified, consistent dataset. It builds upon cleansing by ensuring integrated data adheres to a common model.

• Syntactic Harmonization: Converting date formats (MM/DD/YYYY to YYYY-MM-DD), units (lbs to kg), and character encodings.
• Structural Harmonization: Aligning nested JSON structures to flat relational tables or RDF triples.
• Semantic Harmonization: Mapping 'Customer' from one source to 'Client' in another using an ontology.

The process of determining whether records from different data sources refer to the same real-world entity, a critical step after cleansing to eliminate duplicates across systems. Also known as Identity Resolution or Record Linkage.

• Uses fuzzy matching algorithms (e.g., Levenshtein distance) and probabilistic models.
• Distinguishes between 'J. Smith Corp', 'J Smith Corporation', and 'John Smith Corp'.
• Outputs a Golden Record—a single, authoritative view of each resolved entity for the knowledge graph.

The process of converting data that has multiple possible representations into a single, standard, authoritative form. It is a specific cleansing operation that enforces consistency.

• Text Canonicalization: Converting 'St.', 'Street', and 'St' all to a standard 'Street'.
• URI Canonicalization: Ensuring all references to a resource use the same Internationalized Resource Identifier (IRI) in an RDF knowledge graph.
• Value Canonicalization: Standardizing product codes or country names to a controlled vocabulary.

The process of enhancing, refining, or augmenting raw data with additional context or attributes from external sources to increase its value for the knowledge graph. It often occurs after initial cleansing.

• Geospatial Enrichment: Adding latitude/longitude coordinates to an address.
• Firmographic Enrichment: Appending company industry codes, revenue estimates, or employee count.
• Semantic Enrichment: Linking a product mention to its DBpedia or Wikidata entity URI.

The process of establishing semantic correspondences between the attributes, tables, or classes of two or more heterogeneous data schemas to enable integration. It defines the 'map' for transformation and cleansing rules.

• Attribute Matching: Determining that cust_name in one database is equivalent to customerName in another.
• Type Conversion: Defining that a source STRING field for price must be transformed to a target DECIMAL.
• Relationship Mapping: Aligning a source 'belongs_to' relationship to a target 'hasDepartment' property.

The tracking of data's origins, movements, transformations, and dependencies across its lifecycle. For cleansing, lineage provides critical visibility for auditing changes and debugging quality issues.

• Provenance Tracking: Recording which cleansing rule corrected a specific invalid ZIP code.
• Impact Analysis: Understanding which downstream knowledge graph queries or reports are affected by a change to a deduplication algorithm.
• Compliance Auditing: Demonstrating the steps taken to sanitize personally identifiable information (PII) for regulatory purposes.