TF-IDF

The weight is the product of two factors: Term Frequency (TF) and Inverse Document Frequency (IDF).

• TF: Raw count or logarithmically scaled frequency of term t in document d.
• IDF: Logarithmically scaled inverse fraction of documents containing term t.
• Formula: TF-IDF(t, d) = TF(t, d) * log(N / DF(t))

This multiplication ensures a term is highly relevant only if it appears often locally but rarely globally.

TF-IDF produces a sparse vector for each document, where most dimensions are zero.

• Each unique term in the corpus becomes a dimension.
• The vector length equals the vocabulary size.
• Non-zero values exist only for terms present in the document.
• This contrasts with dense embeddings from neural models, making TF-IDF highly memory-efficient and interpretable for exact keyword matching.

The IDF component naturally penalizes high-frequency function words.

• Words like 'the', 'is', and 'and' appear in almost every document.
• Their DF(t) approaches N, making log(N / DF(t)) approach zero.
• This automatically suppresses stop words without requiring a predefined list.
• The mechanism acts as a statistical noise filter, elevating discriminative content words.

Raw term frequency can be sublinearly scaled to prevent dominant terms from overwhelming the signal.

• Standard TF: 1 + log(raw_count) if count > 0, else 0.
• A term appearing 100 times is not 100x more important than one appearing once.
• This logarithmic dampening accounts for the diminishing marginal relevance of repeated mentions.
• It prevents document length bias from skewing similarity calculations.

BM25 is the probabilistic successor to TF-IDF that refines the weighting scheme.

• Introduces document length normalization to avoid bias toward longer documents.
• Uses a non-linear term saturation function: TF / (k1 * ((1-b) + b * doc_len/avg_len) + TF).
• The k1 and b parameters control saturation speed and length normalization impact.
• BM25 consistently outperforms classic TF-IDF in modern sparse retrieval benchmarks.

Multiple smoothing variants exist to handle edge cases where a term appears in zero documents.

• Standard IDF: log(N / df_t)
• Smooth IDF: log(1 + N / df_t) — prevents division by zero.
• Probabilistic IDF: log((N - df_t) / df_t) — frames rarity as odds.
• TF-ICF adapts this for domain specificity by replacing document count with corpus count across domains.

An adaptation of TF-IDF that replaces Inverse Document Frequency with Inverse Corpus Frequency. While IDF measures rarity across a general background corpus, ICF measures rarity against a domain-specific corpus. This makes TF-ICF superior for identifying terms that are distinctive within a specialized field rather than just rare in general language.

• Formula: TF(t,d) * log(N_c / df(t,c))
• Use Case: Distinguishing domain jargon from general rare words
• Advantage: Prevents common domain terms from being penalized

The Best Match 25 algorithm is the modern evolution of TF-IDF, forming the default sparse retrieval backbone for systems like Elasticsearch. It introduces term saturation (diminishing returns for high TF) and document length normalization to prevent long documents from dominating.

• Saturation: Limits the impact of term frequency using a non-linear function
• Parameters: k1 (term frequency scaling) and b (length normalization)
• Status: Outperforms raw TF-IDF in most full-text search benchmarks

The primary evaluation metric for keyphrase extraction systems. It computes the harmonic mean of precision and recall considering only the top-K predicted keyphrases. This metric penalizes both over-generation and under-generation of key terms.

• Precision@K: How many of the top-K predictions are correct
• Recall@K: How many of the gold-standard keyphrases were captured
• Typical K: 5, 10, or 15 for scientific document evaluation
• Benchmark: Standard metric on the KP20k dataset

A robust data fusion method used to combine the ranked outputs of multiple retrieval or extraction systems, such as merging a TF-IDF sparse list with a dense embedding list. It sums the reciprocal of the rank positions.

• Formula: Score(d) = Σ 1 / (k + rank_i(d))
• Constant k: Typically 60, dampens high-rank impact
• Advantage: No score normalization required; handles disparate score distributions
• Application: Hybrid search combining BM25 and vector similarity

A network representation foundational to graph-based ranking algorithms like TextRank. Nodes represent words or candidate phrases, and edges are weighted by the frequency of co-occurrence within a sliding window of N words.

• Window Size: Typically 2-10 words
• Edge Weight: Pointwise mutual information or raw frequency
• Iteration: Runs PageRank until convergence to score node centrality
• Contrast: Unlike TF-IDF, captures structural relationships between terms rather than corpus-level statistics

A re-ranking algorithm that balances a phrase's relevance to the document against its redundancy with already selected keyphrases. It prevents the extraction of near-duplicate terms that TF-IDF might score highly.

• Formula: MMR = argmax[ λ * Sim(d, c) - (1-λ) * max Sim(c, selected) ]
• Parameter λ: Controls the diversity-relevance trade-off
• Use Case: Applied as a post-processing step after initial TF-IDF scoring
• Goal: Produce a diverse, non-redundant keyphrase set