TF-IDF Vectorization

Measures the raw density of a term within a specific document. It is calculated as the number of times a word appears divided by the total word count of the document, normalizing for document length.

• Raw Count: Simple frequency of term t in document d.
• Normalization: Prevents bias toward longer documents.
• Double Normalization 0.5: Often used to dampen the effect of very high frequencies, preventing a single repeated word from dominating the vector.

Measures the global rarity of a term across the entire corpus. It diminishes the weight of commonly occurring words (like 'the' or 'is') and amplifies the weight of rare, discriminative words.

• Calculation: log(Total Documents / Number of Documents containing term t).
• Stop Word Penalty: Automatically assigns near-zero weight to functional words.
• Specificity Signal: A high IDF indicates a term that is highly specific to a subset of documents.

The final weight is the product of the local frequency and the global rarity: TF-IDF(t,d) = TF(t,d) * IDF(t).

• High Weight: Achieved when a term appears frequently in a specific document but rarely in the rest of the corpus.
• Low Weight: Occurs when the term appears in almost all documents (low IDF) or is absent from the document (zero TF).
• Sparsity: The resulting matrix is mostly zeros, as most documents contain only a small subset of the total vocabulary.

TF-IDF transforms a text corpus into a document-term matrix, where rows represent documents and columns represent unique terms. Each cell contains the TF-IDF weight.

• Dimensionality: The vector length equals the size of the corpus vocabulary (often thousands of dimensions).
• Geometric Interpretation: Documents become points in a high-dimensional space, allowing for cosine similarity comparisons.
• Semantic Limitation: This is a 'bag-of-words' model; it ignores word order and context, treating sentences as an unordered collection of tokens.

A variant that applies a logarithmic function to the raw term frequency: TF = 1 + log(TF_raw).

• Saturation Effect: Prevents a term that appears 20 times from being treated as 20 times more important than a term appearing once.
• Relevance Modeling: Better aligns with the psychological observation that the perception of relevance does not increase linearly with repetition.
• Implementation: Commonly available as a parameter in libraries like Scikit-learn's TfidfVectorizer.

Once documents are vectorized, cosine similarity measures the cosine of the angle between two TF-IDF vectors, ignoring magnitude differences caused by document length.

• Range: Scores range from 0 (orthogonal, no shared terms) to 1 (identical direction, highly similar).
• Near-Duplicate Threshold: A score above 0.8 typically indicates a near-duplicate pair.
• Efficiency: Sparse vector multiplication makes this computationally feasible even for large corpora, forming the backbone of canonicalization pipelines.

The primary metric used to compare TF-IDF vectors. It measures the cosine of the angle between two non-zero vectors in a multi-dimensional space, producing a value between -1 and 1. A score of 1 indicates identical orientation (semantically similar documents), while 0 indicates orthogonality (no similarity).

• Formula: cos(θ) = (A · B) / (||A|| × ||B||)
• Unaffected by document length, only term distribution
• Core mechanism for near-duplicate detection and document clustering

The raw count of how often a term appears in a document, normalized by document length to prevent bias toward longer texts. Common variants include:

• Raw count: Simple frequency of term t in document d
• Log normalization: 1 + log(freq) to dampen the effect of high-frequency terms
• Double normalization 0.5: 0.5 + 0.5 × (freq / max_freq_in_doc)

TF captures the local importance of a term within a single document.

A measure of how much information a term provides across the entire corpus. It diminishes the weight of terms that appear in many documents and amplifies rare, discriminative terms.

• Formula: IDF(t) = log(N / df(t)) where N is total documents and df(t) is the number of documents containing term t
• Stop words like 'the' or 'and' receive near-zero IDF weights
• The inverse relationship penalizes ubiquitous terms and rewards distinctive vocabulary

The underlying text representation assumption behind TF-IDF. Documents are treated as an unordered collection of words, discarding grammar, word order, and syntax. Each document becomes a vector where each dimension corresponds to a unique term in the vocabulary.

• Simplifies text to a sparse vector of term frequencies
• Enables efficient matrix operations across large corpora
• Limitation: loses semantic relationships and context between words
• Foundation for more advanced techniques like n-grams and embeddings

TF-IDF vectors are inherently sparse—most dimensions are zero because any single document uses only a tiny fraction of the corpus vocabulary. Efficient storage and computation rely on sparse matrix formats:

• Compressed Sparse Row (CSR): Stores only non-zero values and their column indices
• Dictionary of keys (DOK): Hash-based representation for incremental construction
• Enables processing of corpora with millions of unique terms without memory explosion
• Critical for production-scale duplicate detection pipelines

A two-dimensional matrix where rows represent documents and columns represent unique terms from the entire corpus vocabulary. Each cell contains the TF-IDF weight for that term-document pair.

• Dimensions: D (documents) × V (vocabulary size)
• Used as input for dimensionality reduction techniques like SVD or NMF
• Forms the basis for latent semantic analysis (LSA)
• Can be transposed to find similar terms based on document co-occurrence patterns