Feature Engineering

These are the foundational numeric measures that quantify the likeness between two candidate records. They are the direct inputs to matching models.

• String Similarity: Metrics like Levenshtein distance, Jaccard similarity (for token sets), and cosine similarity (for text embeddings) compare names, addresses, and other text fields.
• Numeric Similarity: Measures like absolute difference or relative difference for dates (e.g., birth dates), amounts, or IDs.
• Categorical Similarity: Exact match, partial match, or hierarchical match for fields like product categories or industry codes.

Creating new attributes from raw data provides stronger, more abstract signals for disambiguation.

• Phonetic Encoding: Converting strings to codes based on pronunciation (e.g., Soundex, Metaphone) enables matching of 'Jon' and 'John'.
• Temporal Features: Calculating age from birth date, duration between events, or extracting day-of-week from timestamps.
• Geospatial Features: Computing the haversine distance between addresses or mapping to administrative regions (city, state).
• Composite Features: Creating features from the interaction of others, like a flag for 'name matches but cities are >100 miles apart'.

Not a model feature per se, but a critical pre-computation. Blocking keys are hashes or signatures created to place potentially matching records into the same candidate set, drastically reducing pairwise comparisons.

• Purpose: Transform high-dimensional comparison problem into manageable sub-problems.
• Common Techniques: Using the first N letters of a name, a phonetic code, or a geographic hash. Locality-Sensitive Hashing (LSH) is an advanced method for creating keys that preserve similarity.
• Trade-off: Overly restrictive keys miss true matches (low recall); overly permissive keys create bloated candidate sets (low efficiency).

When resolving entities within a network or knowledge graph, features can be extracted from the relational structure.

• Neighborhood Features: Attributes of connected entities, such as 'the average transaction amount of connected accounts'.
• Path Features: The existence or length of a path between two candidate entities in a pre-existing graph.
• Community Features: Whether two records belong to the same detected cluster or community within a larger network. These features help resolve entities using collective relationships, not just attribute similarity.

After creating a broad set of candidate features, identifying the most informative ones is crucial for model performance and interpretability.

• Methods: Techniques like correlation analysis, recursive feature elimination, or using built-in importance scores from tree-based models (e.g., Gini importance from Random Forest).
• Goal: Remove redundant or noisy features to reduce overfitting, speed up training, and simplify the model.
• Outcome: A pared-down set of features that provides the highest predictive power for distinguishing matches from non-matches.

Real-world data is imperfect. Engineering features robust to missing values and errors is a core challenge.

• Imputation Strategies: For missing values, use global statistics (mean/mode), model-based imputation, or a dedicated 'missing' indicator flag.
• Noise-Tolerant Similarity: Using fuzzy matching techniques or similarity metrics that are less sensitive to minor typos and variations.
• Validation: Creating features that signal data quality, such as 'number of null fields in record' or 'format conformity score', which can be used to weight the confidence of other features.

Raw text data is often inconsistent. This foundational step creates a uniform baseline for comparison.

Key operations include:

• Lowercasing: Converting 'Apple Inc.', 'APPLE INC.', and 'apple inc.' to 'apple inc.'.
• Whitespace & Punctuation Removal: Stripping extra spaces and standardizing delimiters.
• Stop Word Removal: Removing common words like 'the', 'inc.', or 'llc' if they don't aid discrimination.
• Unicode Normalization: Ensuring characters like 'café' and 'café' are represented identically (NFD or NFC form).

Example: The strings 'St. Jude Children's Hospital', 'St Jude Childrens Hospital', and 'Saint Jude Children’s Hospital' are normalized to a comparable form before similarity calculation.

Algorithms convert words into codes based on pronunciation to match names that sound alike but are spelled differently, handling common transcription errors.

Common Algorithms:

• Soundex: A classic algorithm that groups consonants. 'Robert' and 'Rupert' both encode to 'R163'.
• Metaphone & Double Metaphone: More advanced English phonetic algorithms. 'Smith', 'Smyth', and 'Schmidt' may map to the same code ('SM0' or 'XMT').
• NYSIIS: The New York State Identification and Intelligence System, often used for governmental records.

Use Case: Linking customer records where 'Katherine' may be entered as 'Catherine', 'Kathryn', or 'Cathryn'. Phonetic features provide a robust signal beyond edit distance.

Dates and timestamps are rich sources of features but must be decomposed into comparable, meaningful components.

Extracted features often include:

• Absolute Differences: Days between 'date_of_birth' fields.
• Cyclical Encoding: Transforming dates into sine/cosine waves to capture periodicity (e.g., day-of-week, month). This ensures December and January are close in vector space.
• Age Calculation: Deriving age from a birth date relative to a reference date (e.g., transaction date).
• Event Intervals: Time between 'account_created' and 'first_purchase' dates.

Example: For patient record linkage, the absolute difference in date_of_birth is a strong negative signal, while similarity in cyclical 'day_of_week_of_first_visit' might be a weak positive signal.

Categorical variables like 'job_title' or 'product_category' require transformation into numerical features a model can use.

Common Techniques:

• One-Hot Encoding: Creating binary columns for each category. Can become high-dimensional for variables with many values.
• Target Encoding: Replacing a category with the average value of the target variable (e.g., match probability) for that category. Risks data leakage; must be calculated carefully.
• Frequency Encoding: Replacing a category with its frequency of occurrence in the dataset. Common categories get a high value.
• Learned Embeddings: Using a model to learn a dense, low-dimensional vector representation for each category during training.

Use Case: Encoding 'company_industry' (e.g., 'Technology', 'Finance', 'Healthcare') into a feature that captures semantic similarity.

Individual attribute similarities (names, addresses) are combined into a single, powerful feature vector for a classifier.

Process:

1. Calculate a base similarity score for each attribute pair (e.g., Jaccard for name tokens, Levenshtein for street number).
2. Apply non-linear transformations like scaling or sigmoid functions to normalize scores.
3. Create interaction features by multiplying or averaging scores (e.g., name_similarity * address_similarity).
4. Derive aggregate statistics like max(similarity_scores) or number_of_scores_above_threshold.

Example Feature Vector for a Pair of Customer Records: [name_jaccard=0.8, addr_levenshtein_norm=0.2, email_exact_match=1.0, (name_jaccard * email_match)=0.8, max_similarity=1.0]

When resolving entities within a larger network, features from the graph structure itself provide critical context.

Extracted Features Include:

• Neighborhood Overlap: Jaccard similarity of the sets of nodes connected to two candidate entities.
• Degree Centrality: The number of connections each node has. High-degree entities (e.g., a common company name) may require stronger evidence for a match.
• Path-Based Features: The existence or length of a shortest path between two records in a pre-existing graph.
• Community Assignment: Whether two records belong to the same detected cluster or community in the graph.

Use Case: In academic publication deduplication, two author records are more likely to be the same person if they co-author with a highly overlapping set of other researchers (neighborhood overlap).

The process of identifying and retaining the most relevant, non-redundant features from a dataset to improve model performance and reduce computational cost. For entity resolution, this involves determining which attributes (e.g., name, address, phone number) are most discriminative for matching entities.

• Methods include filter methods (correlation scores), wrapper methods (recursive feature elimination), and embedded methods (L1 regularization).
• Goal: Reduce overfitting, speed up training, and enhance model interpretability by removing noisy or irrelevant data.

The application of mathematical functions to modify the scale, distribution, or relationship of features. This is critical for making data compatible with algorithm assumptions and improving similarity calculations in entity resolution.

• Common techniques:
  • Normalization: Scaling features to a range (e.g., 0 to 1).
  • Standardization: Transforming features to have a mean of 0 and standard deviation of 1 (Z-score).
  • Log/Power Transforms: Reducing skewness in highly non-normal distributions (e.g., transaction counts).
• Impact: Ensures features contribute equally to distance metrics like cosine similarity.

The technique of creating new, more informative features by combining or decomposing raw variables. It reduces dimensionality while preserving essential information.

• For text-based entity resolution:
  • Generating n-gram features from names or addresses.
  • Creating phonetic codes (Soundex, Metaphone) for matching similar-sounding strings.
• Advanced methods:
  • Principal Component Analysis (PCA): Creates uncorrelated components.
  • Autoencoders: Neural networks that learn compressed representations.
• Creates compact, semantically rich inputs for matching models.

A method for converting categorical variables into a binary vector format that can be provided to machine learning algorithms. Each category value becomes a new binary (0/1) feature.

• Example: A 'Country' feature with values {USA, Canada, UK} becomes three new features: is_USA, is_Canada, is_UK.
• Use in Entity Resolution: Encodes categorical match keys (e.g., gender codes, product types) for probabilistic models. It prevents the model from incorrectly inferring ordinal relationships where none exist.
• Alternative: Label Encoding assigns a unique integer to each category, but is only suitable for tree-based models.

A synthetic feature formed by multiplying (or combining) two or more existing features. It enables a model to learn non-linear interactions between features that would otherwise be treated independently.

• Example: For customer matching, crossing City and Postal_Code_First3 might create a powerful signal for geographic clustering.
• Impact: Crucial for linear models (like logistic regression) to capture complex decision boundaries without manual rule creation. In entity resolution, it can help models learn that certain attribute combinations are highly indicative of a match (e.g., a specific last name with a rare phone prefix).

A metric that quantifies the contribution of each input feature to the predictions of a trained machine learning model. It identifies which features the model relies on most.

• Calculation Methods:
  • Tree-based: Gini impurity reduction or mean decrease in accuracy (e.g., from Random Forest, XGBoost).
  • Model-agnostic: SHAP (SHapley Additive exPlanations) values assign each feature an importance value for a specific prediction.
• Application: In entity resolution pipelines, analyzing feature importance validates the engineering process, showing whether derived features (like similarity scores) are being utilized effectively. It guides iterative refinement of the feature set.