YAKE

YAKE computes five distinct statistical features for each candidate term to determine its relevance:

• Casing: Rewards capitalized and acronym terms, penalizing lowercase words that are less likely to be important
• Term Position: Boosts terms appearing early in the document, where key concepts are typically introduced
• Term Frequency Normalization: Balances raw frequency against the frequency of sub-terms to prevent common words from dominating
• Term Relatedness to Context: Measures how many different terms co-occur with the candidate, penalizing stopword-like behavior
• Sentence Frequency: Rewards terms that appear across multiple sentences, indicating broader topical relevance

These features are combined into a single score, with lower values indicating higher keyword relevance.

Unlike TF-IDF which requires a reference corpus to compute inverse document frequency, YAKE operates entirely on a single input document. This makes it ideal for:

• Processing isolated documents where no corpus exists
• Real-time applications with streaming text
• Edge deployments with limited storage
• Privacy-sensitive contexts where external data access is restricted

The algorithm builds its statistical model solely from the internal structure and distribution of terms within the provided text, eliminating corpus dependency entirely.

YAKE constructs a co-occurrence graph using a sliding window of configurable size (default: 10 words) across the document. This captures local semantic relationships:

• Words appearing together frequently within the window receive higher relatedness scores
• The window size can be tuned: smaller windows capture tight phrases, larger windows capture broader topical associations
• This graph-based approach allows YAKE to identify multi-word expressions without relying on external phrase dictionaries

The co-occurrence statistics feed directly into the term relatedness feature, distinguishing content-bearing terms from functional words.

YAKE generates candidate keyphrases by extracting contiguous n-grams up to a maximum length (default: 3 words). The process:

• Splits text on stopwords and punctuation to identify phrase boundaries
• Generates all possible 1-gram, 2-gram, and 3-gram sequences from these segments
• Scores each candidate using the combined statistical features
• Applies deduplication to remove redundant overlapping phrases

The stopword-delimited segmentation naturally identifies phrase boundaries without requiring part-of-speech tagging or syntactic parsing, keeping the algorithm fast and language-agnostic.

YAKE achieves broad language support through its reliance on statistical patterns rather than linguistic rules. Key design choices:

• Uses a configurable stopword list as the only language-specific resource
• Avoids dependency on POS taggers, parsers, or pretrained embeddings
• Works effectively on languages with available stopword lists, including English, Spanish, French, German, Italian, and many others
• The statistical features (casing, position, frequency) are language-agnostic

This makes YAKE suitable for multilingual document processing pipelines where maintaining language-specific NLP toolchains would be impractical.

After scoring all candidate n-grams, YAKE applies a deduplication algorithm to remove redundant overlapping phrases:

• If a longer phrase and its sub-phrase both appear as candidates, the algorithm compares their scores
• The higher-scoring variant is retained while the lower-scoring duplicate is eliminated
• This prevents both 'machine learning' and 'learning' from appearing in the final output when they represent the same concept
• Final keyphrases are ranked by their composite score in ascending order (lower is better)

The result is a clean, non-redundant list of the most salient keyphrases extracted from the document.

The foundational statistical measure for keyword importance. It calculates a term's weight by multiplying Term Frequency (how often a word appears in a document) by Inverse Document Frequency (how rare it is across a corpus). Unlike YAKE, TF-IDF requires a background corpus to compute IDF, making it less suitable for single-document extraction. It excels at identifying discriminative single words but often fails to capture multi-word phrases without additional n-gram heuristics.

An unsupervised, domain-independent algorithm that extracts keyphrases by analyzing word co-occurrence. It splits text at stopwords to generate candidate phrases, then scores them based on the degree and frequency of constituent word co-occurrences within a graph. Like YAKE, it operates on a single document without external corpora, but relies solely on graph centrality rather than YAKE's multi-feature statistical model.

A graph-based ranking algorithm adapted from Google's PageRank. It builds a word co-occurrence network where nodes are lexical units and edges represent co-occurrence within a window. The algorithm iteratively scores nodes until convergence, selecting the highest-ranked as keyphrases. Unlike YAKE's statistical feature approach, TextRank relies purely on the structural properties of the text graph and requires no labeled training data.

A method leveraging BERT embeddings to extract keyphrases by measuring cosine similarity between candidate phrases and the full document embedding. It represents a shift from statistical features to dense semantic representations. While YAKE uses surface-level features like casing and position, KeyBERT captures deeper contextual meaning, often producing more semantically coherent phrases at the cost of higher computational requirements.

A re-ranking algorithm that balances a phrase's relevance to the document against its redundancy with already selected keyphrases. Often used as a post-processing step after YAKE or other extractors to diversify results. MMR prevents the top-K list from being dominated by semantically identical phrases, ensuring broader topical coverage in the final keyphrase set.

The core process of assigning numerical weights to potential keyphrases. YAKE's scoring model uniquely combines five statistical features: casing (capitalization signals importance), position (early occurrence boosts weight), frequency (normalized term count), relatedness to context (co-occurrence with frequent terms), and sentence frequency (how often a term appears across different sentences). This multi-feature ensemble distinguishes it from simpler frequency-based methods.