KeyBERT

The core mechanism relies on cosine similarity between the document embedding and candidate phrase embeddings. After generating a document-level vector using a Sentence Transformer model, KeyBERT encodes each candidate n-gram and ranks them by their proximity to the document vector. This ensures that the selected keyphrases capture the central theme rather than just frequent terms.

To avoid redundancy, KeyBERT integrates Maximal Marginal Relevance. After the initial ranking, MMR iteratively selects phrases that are highly similar to the document but dissimilar to already chosen keyphrases. This balances informativeness with diversity, ensuring the final output covers distinct aspects of the text rather than repeating the same concept.

KeyBERT uses scikit-learn's CountVectorizer to generate candidate phrases. By default, it extracts n-grams within a specified range, often filtering by part-of-speech patterns to retain noun phrases. This statistical approach to phrase candidate generation is language-agnostic and requires no external linguistic pipelines, keeping the library lightweight and dependency-minimal.

While defaulting to all-MiniLM-L6-v2, KeyBERT supports any Sentence Transformer model. Users can inject domain-specific models like allenai-specter for scientific text or multilingual models like paraphrase-multilingual-MiniLM-L12-v2. This flexibility allows the extraction quality to scale with the sophistication of the underlying semantic similarity model.

KeyBERT supports seed-guided extraction where a user provides a list of seed words. The algorithm then biases the selection towards keyphrases that are semantically similar to both the document and the seed terms. This is particularly useful for extracting domain-specific jargon or aligning extracted tags with a pre-existing controlled vocabulary or taxonomy.

The standard evaluation metric for keyphrase extraction. It computes the harmonic mean of precision and recall for the top-K predicted keyphrases against a gold-standard set.

• Precision@K: How many of the top-K are correct?
• Recall@K: How many of the total correct keyphrases were found in the top-K?
• Usage: F1@5 and F1@10 are common benchmarks on datasets like KP20k.

The process of assigning a numerical weight to each phrase before ranking. KeyBERT uses semantic similarity, but other methods use statistical features:

• TF-IDF: Weights terms by rarity across documents.
• YAKE: Uses 5 statistical features like casing and position.
• Position Bias: Words in the title or abstract often score higher.

The crucial first step before embedding. This process extracts potential n-grams from text, often using Part-of-Speech (POS) tagging to filter for noun phrases (e.g., adjectives + nouns). KeyBERT typically uses CountVectorizer with n-gram ranges to generate these candidates before embedding them.

A technique to improve robustness by combining the ranked outputs of multiple algorithms. You might fuse KeyBERT's semantic scores with TextRank's graph-based scores using Reciprocal Rank Fusion (RRF). This ensures that a phrase must be both statistically salient and semantically central to rank highly.