TF-IDF

Measures the raw frequency of a term within a specific document. The core assumption is that a word appearing more often is more relevant to the document's topic.

• Raw Count: The simplest form is the number of times term t appears in document d.
• Normalization: Often normalized by document length to prevent bias toward longer documents (e.g., dividing by total word count).
• Logarithmic Scaling: A common variant uses log(1 + ft,d) to dampen the effect of very high frequencies.

Quantifies the informativeness of a term across the entire document collection. It downweights common words that appear in many documents and upweights rare, discriminative terms.

• Formula: IDF(t) = log(N / dft), where N is the total number of documents and dft is the number of documents containing term t.
• Stop Word Effect: Common words like 'the' or 'is' receive an IDF score near zero, effectively neutralizing their impact.
• Specificity Signal: A high IDF value indicates a term is a strong, rare keyword for a specific subset of documents.

The final weight is the product of the term frequency and inverse document frequency components. This composite score balances local frequency with global rarity.

• Core Calculation: TF-IDF(t, d) = TF(t, d) * IDF(t).
• High Score: Achieved when a term has a high frequency in a specific document (high TF) but a low document frequency across the collection (high IDF).
• Vector Representation: The TF-IDF score for each unique term becomes a dimension in a sparse vector representing the document, enabling direct comparison via cosine similarity.

While TF-IDF is foundational, its linear term frequency scaling can be problematic. A term appearing 100 times is rarely 10 times more relevant than one appearing 10 times. Modern ranking functions address this.

• Non-Linear Saturation: The BM25 algorithm introduces a saturation function, where the gain from additional term occurrences plateaus.
• k1 Parameter: In BM25, the k1 parameter controls the shape of this saturation curve, preventing term frequency from dominating the score.
• Legacy Impact: TF-IDF remains the conceptual ancestor of BM25's term weighting component, which is the default relevance scoring in engines like Elasticsearch.

TF-IDF weights are the primary data stored in an inverted index, the core data structure enabling fast, large-scale lexical search.

• Postings List: The inverted index maps each term to a postings list containing document IDs and their corresponding TF-IDF weights.
• Computational Efficiency: At query time, the engine only needs to fetch the postings lists for the query terms, making retrieval extremely fast compared to scanning all documents.
• Lexical Matching: This process is the foundation of sparse retrieval, where relevance is determined by explicit word overlap rather than dense semantic vectors.

The primary limitation of TF-IDF and other bag-of-words models is the inability to handle semantic similarity without exact word overlap.

• Synonym Failure: A document about 'automobiles' will not be retrieved for a query for 'cars' unless a synonym filter is manually configured in the analyzer.
• Polysemy: The model cannot distinguish between different meanings of the same word (e.g., 'bank' as a financial institution vs. a river bank).
• Solution: This limitation motivates dense retrieval models, which use neural networks to map semantically similar words to nearby points in a continuous vector space.