TF-IDF Vectorization

Measures the raw frequency of a term within a specific document, normalized by document length to prevent bias toward longer texts.

• Formula: TF(t,d) = (Number of times term t appears in document d) / (Total number of terms in document d)
• Purpose: Captures local importance—a term appearing 10 times in a 100-word document has higher TF than one appearing 10 times in a 10,000-word document
• Variants: Raw count, logarithmic normalization (1 + log(count)), and augmented frequency for length normalization
• Limitation: High TF alone cannot distinguish between common stop words and meaningful discriminators

Suppresses the weight of terms that appear frequently across the entire corpus, isolating discriminative terms that are rare and therefore more informative.

• Formula: IDF(t) = log(Total number of documents / Number of documents containing term t)
• Mechanism: A term appearing in every document receives an IDF of 0, effectively neutralizing its contribution
• Example: In a corpus of loan applications, the word 'applicant' appears universally and gets low IDF; 'bankruptcy' appears rarely and gets high IDF
• Smoothing: Variants add 1 to the denominator to prevent division by zero for out-of-vocabulary terms

Combines TF and IDF multiplicatively to produce a sparse, high-dimensional vector where each dimension corresponds to a unique term in the corpus vocabulary.

• Final Weight: TF-IDF(t,d) = TF(t,d) × IDF(t)
• Output: Each document becomes a vector of length |V| (vocabulary size), where most values are zero
• Sparsity: Typical document vectors are over 99% sparse, making TF-IDF computationally efficient for large corpora
• Interpretation: High TF-IDF scores indicate terms that are both frequent in the document and rare across the corpus—strong semantic signals

Uses the cosine of the angle between TF-IDF vectors to measure document similarity, ignoring magnitude differences caused by varying document lengths.

• Formula: cos(θ) = (A · B) / (||A|| × ||B||)
• Range: Scores from -1 (completely dissimilar) to 1 (identical); in practice, TF-IDF vectors are non-negative, yielding scores between 0 and 1
• Application: In synthetic identity detection, cosine similarity between application text fields flags duplicate or templated submissions
• Advantage: Magnitude-invariant—two applications with identical wording but different lengths receive a perfect similarity score

Applies a logarithmic dampening function to raw term frequency, preventing a term appearing 20 times from having 20× the influence of a term appearing once.

• Transformation: TF_scaled = 1 + log(raw_TF) when raw_TF > 0, else 0
• Rationale: The informational gain from seeing a term a second time is far greater than seeing it a twentieth time
• Impact: Reduces the dominance of repetitive boilerplate language in identity documents while preserving the signal of rare discriminative terms
• Use Case: Critical when analyzing lengthy customer correspondence or legal entity descriptions where key terms may repeat

Extends the basic unigram model to capture multi-word phrases and local word ordering, preserving context that single-word tokenization loses.

• Bigram Example: 'New York' treated as a single token rather than independent 'new' and 'york' terms
• Trade-off: Vocabulary size explodes combinatorially—a 10,000-word unigram vocabulary can produce millions of bigram features
• Filtering: Typically combined with minimum frequency thresholds or chi-squared feature selection to retain only statistically significant n-grams
• Fraud Relevance: Captures templated phrases like 'same day funding' or 'no credit check' that signal coordinated synthetic identity applications

A metric measuring the cosine of the angle between two non-zero vectors in a multi-dimensional space. In TF-IDF pipelines, cosine similarity quantifies the semantic proximity between two documents by comparing their weighted term vectors.

• Score ranges from -1 (opposite) to 1 (identical), with 0 indicating orthogonality
• Computationally efficient for sparse, high-dimensional TF-IDF vectors
• Commonly used to detect duplicate or near-duplicate identity records in entity resolution workflows

A string comparison technique that calculates the probability of two text strings being a match, tolerating typographical errors, abbreviations, and formatting inconsistencies. Unlike TF-IDF which operates on full documents, fuzzy matching targets individual string fields like names and addresses.

• Levenshtein Distance: Measures minimum single-character edits required to transform one string into another
• Jaro-Winkler Similarity: Optimized for short strings like personal names, giving higher scores to strings matching from the beginning
• Often combined with TF-IDF in hybrid identity resolution pipelines

The statistical methodology for joining records from two or more datasets that are believed to correspond to the same entity when a common unique identifier is absent. TF-IDF vectorization serves as a feature engineering step within record linkage systems.

• Deterministic linkage: Exact matching on transformed keys
• Probabilistic linkage: Uses the Fellegi-Sunter model to calculate match probabilities based on agreement patterns
• TF-IDF vectors feed into blocking and scoring stages to reduce the quadratic comparison problem

A performance optimization technique that partitions large datasets into mutually exclusive blocks using a common attribute, drastically reducing the number of record pair comparisons required. Without blocking, comparing every record pair results in O(n²) complexity.

• TF-IDF vectors can be used to generate locality-sensitive hashing blocks
• Common blocking keys include phonetic encodings like Soundex or n-gram signatures
• Essential for scaling identity resolution to millions of records in financial fraud detection systems

The process of building a structured network of real-world entities, their attributes, and the semantic relationships between them. TF-IDF vectorization contributes to the entity extraction and alignment phase of knowledge graph construction.

• TF-IDF helps identify salient terms that define entity nodes
• Combined with named entity recognition to populate graph attributes
• Provides a unified, deduplicated identity view for synthetic identity detection
• Enables link prediction algorithms to infer hidden fraudulent relationships

A cryptographic protocol enabling the matching of records across disparate databases without revealing plaintext personally identifiable information (PII). TF-IDF vectors can be encoded into privacy-preserving representations for secure comparison.

• Bloom filter encoding: Converts TF-IDF-weighted tokens into irreversible bit arrays for secure fuzzy matching
• Enables collaborative fraud detection across institutions without exposing customer data
• Critical for compliance with K-anonymity and data sovereignty requirements in financial services