TF-IDF Vectorization

Measures how often a term appears within a specific document, normalized by document length to prevent bias toward longer texts.

• Raw Count: Simple tally of occurrences
• Log Normalization: 1 + log(tf) dampens the effect of high-frequency words
• Double Normalization: Further adjusts for document length variance

A word appearing 10 times in a 100-word document has a higher TF than one appearing 10 times in a 10,000-word document.

Quantifies how rare or common a term is across the entire document collection, suppressing the weight of frequently occurring words that carry little discriminatory power.

• Formula: log(N / df) where N is total documents and df is the number of documents containing the term
• Smoothing: Adding 1 to the denominator prevents division by zero

Common words like "the" or "and" receive near-zero IDF scores, while rare domain-specific terms receive high weights.

TF-IDF produces high-dimensional but sparse vectors where most values are zero, as any single document contains only a tiny fraction of the total vocabulary.

• A corpus with 100,000 unique terms yields 100,000-dimensional vectors
• A typical document may have fewer than 1,000 non-zero entries
• Efficient storage via compressed sparse row (CSR) or dictionary-of-keys formats

This sparsity enables fast similarity computations using optimized linear algebra libraries.

Once documents are vectorized, cosine similarity measures the angle between two TF-IDF vectors, providing a robust metric for document relatedness that ignores magnitude differences.

• Range: 0 (orthogonal, unrelated) to 1 (identical direction)
• Formula: cos(θ) = (A · B) / (||A|| × ||B||)
• Invariant to document length, unlike Euclidean distance

This is the foundation for search ranking, document clustering, and content recommendation systems.

Raw term frequency can overemphasize words that appear many times in a document. Sublinear scaling applies a logarithmic function to compress the TF component.

• A word appearing 100 times is not 100× more important than one appearing once
• tf = 1 + log(raw_tf) when raw_tf > 0, otherwise 0
• Prevents keyword-stuffed documents from dominating search results

This variant is the default in scikit-learn's TfidfVectorizer and is widely adopted in production systems.

TF-IDF vectorization can operate on multi-word sequences (n-grams) rather than individual tokens, capturing phrase-level semantics that unigrams miss.

• Unigrams: "machine", "learning"
• Bigrams: "machine learning", "learning algorithm"
• Character n-grams: Robust to typos and morphological variations

Combining unigrams and bigrams dramatically increases dimensionality but captures critical context like distinguishing "not good" from "good."

The raw count of a term's occurrences within a specific document, often normalized by document length to prevent bias toward longer texts. TF answers: 'How often does this word appear here?'

• Raw TF: Simple count of term t in document d
• Log normalization: 1 + log(tf) dampens the effect of high-frequency words
• Double normalization: Divides by the max frequency of any term in the document
• Example: In a 100-word document where 'algorithm' appears 3 times, the raw TF is 3, but the normalized TF might be 0.03

A measure of how much information a term provides across a corpus. IDF diminishes the weight of terms that appear in many documents and increases the weight of rare, discriminative terms.

• Formula: IDF(t) = log(N / df(t)) where N is total documents and df(t) is the number of documents containing term t
• Smoothing: Adding 1 to the denominator prevents division by zero for unseen terms
• Interpretation: A word appearing in every document gets an IDF of 0, contributing nothing to the final score

A simplifying representation where text is treated as an unordered collection of words, discarding grammar and word order. TF-IDF is a weighting scheme applied on top of the bag-of-words model.

• Sparsity: Most document vectors are sparse, with non-zero values only for the vocabulary subset present
• Limitation: Loses all semantic context and word sequence information
• Advantage: Computationally efficient and surprisingly effective for document classification and retrieval tasks

The primary metric for comparing two TF-IDF vectors, measuring the cosine of the angle between them rather than their magnitude. This normalizes for document length.

• Range: 0 (orthogonal, no similarity) to 1 (identical direction)
• Formula: cos(θ) = (A · B) / (||A|| × ||B||)
• Use case: Ranking search results by relevance to a query vector
• Contrast: Unlike Euclidean distance, cosine similarity ignores document length, focusing purely on term proportion overlap

A modern alternative to sparse TF-IDF vectors. Word2Vec, GloVe, and BERT embeddings map words into dense, low-dimensional vector spaces where semantic relationships are preserved as geometric distances.

• Dimensionality: Typically 100-768 dimensions vs. TF-IDF's vocabulary-sized vectors (often 10,000+)
• Semantic capture: 'King - Man + Woman ≈ Queen' is possible, which TF-IDF cannot represent
• Trade-off: Embeddings require pre-training on massive corpora; TF-IDF is corpus-specific and interpretable

A probabilistic relevance function that evolved from TF-IDF, forming the default scoring mechanism in search engines like Elasticsearch and Lucene. BM25 introduces term saturation and document length normalization.

• Key improvement: TF contribution saturates—seeing a term 10 times isn't 10x as relevant as seeing it once
• Parameters: k1 controls term frequency saturation; b controls document length normalization
• Adoption: Outperforms vanilla TF-IDF in most information retrieval benchmarks