Keyphrase Extraction

Algorithms like RAKE (Rapid Automatic Keyword Extraction) and YAKE operate on the principle that keyphrases contain words with high term frequency but low document frequency across the corpus. These systems first identify candidate phrases by splitting text at phrase delimiters and stop words, then score candidates using statistical features:

• Term Frequency: How often a word appears in the document
• Term Relatedness: Degree of co-occurrence with other candidate terms
• Casing and Position: Capitalization patterns and sentence-level placement

Unlike pure TF-IDF, these unsupervised methods require no training data, making them ideal for domain-agnostic applications where labeled corpora are unavailable.

TextRank adapts Google's PageRank algorithm to natural language by constructing a co-occurrence graph where nodes represent candidate words and edges represent lexical proximity within a sliding window. The algorithm iteratively computes node importance until convergence:

• Words that co-occur with many other high-scoring words receive elevated scores
• The top-ranked words are assembled into multi-word keyphrases based on adjacency in the original text
• Variants like SingleRank and TopicRank extend this by incorporating topical clustering

This approach excels at identifying latent semantic relationships that pure frequency-based methods miss, particularly in longer documents with complex thematic structure.

Before scoring, extraction systems must generate a clean set of candidate phrases. This preprocessing pipeline typically includes:

• Stop word removal: Eliminating function words that carry minimal semantic weight
• Part-of-speech tagging: Filtering for noun phrases using patterns like (adjective)*(noun)+
• N-gram extraction: Generating contiguous sequences of 1-3 tokens
• Phrase deduplication: Removing overlapping or redundant candidates

The quality of this filtering directly impacts downstream precision. Overly aggressive filtering removes valid keyphrases; overly permissive filtering introduces noise that dilutes ranking accuracy.

Modern systems augment statistical methods with contextualized embeddings from transformer models like BERT. Rather than relying solely on surface-level co-occurrence, these approaches:

• Encode candidate phrases and the full document into a shared vector space
• Compute cosine similarity between each candidate embedding and the document embedding
• Rank phrases by their semantic alignment with the document's core themes

This technique captures paraphrastic relationships—recognizing that 'machine learning' and 'neural network training' are semantically related even when they don't co-occur lexically. Systems like KeyBERT and EmbedRank exemplify this paradigm.

A common failure mode in keyphrase extraction is redundancy—returning multiple near-identical phrases like 'deep learning model' and 'deep learning algorithm'. Maximum Marginal Relevance (MMR) addresses this by iteratively selecting phrases that balance:

• Relevance: High similarity to the document's overall content
• Novelty: Low similarity to already-selected keyphrases

The MMR objective function applies a penalty term controlled by a diversity parameter λ. When λ = 0, the system prioritizes pure relevance; when λ = 1, it maximizes diversity. Production systems typically use λ ≈ 0.6 to ensure the final keyphrase set covers distinct facets of the document's subject matter.

When labeled training data is available, keyphrase extraction can be framed as a sequence labeling task using models like BiLSTM-CRF or fine-tuned transformers. These systems:

• Tokenize the document and assign BIO tags (Beginning, Inside, Outside) to each token
• Learn to recognize keyphrase boundaries based on contextual features
• Achieve higher precision on domain-specific corpora where statistical methods struggle

Supervised methods excel at identifying domain jargon and multi-word expressions that don't follow generic statistical patterns. However, they require substantial annotated data and may not generalize across domains without fine-tuning.

A lightweight, unsupervised extraction method that relies on statistical text features rather than external corpora.

• Considers casing, word position, frequency, and relatedness
• Computes a score inversely proportional to keyword relevance
• Handles multilingual documents without language-specific tuning
• Outperforms TF-IDF on short texts like abstracts and news articles

YAKE is particularly effective for single-document extraction where no corpus-level statistics are available.

A graph-based ranking algorithm adapted from PageRank for natural language processing tasks.

• Builds a co-occurrence graph where words are nodes and edges represent adjacency
• Runs the PageRank algorithm to identify the most central and important terms
• Naturally handles multi-word keyphrases by post-processing adjacent high-scoring words
• Requires no labeled training data

TextRank excels at identifying thematic keywords that capture the core subject of a document through structural importance rather than raw frequency.

A statistical measure that evaluates how important a term is to a document within a larger corpus.

• Term Frequency (TF): How often a word appears in a document
• Inverse Document Frequency (IDF): How rare or common the word is across all documents
• The product TF × IDF penalizes common words while rewarding distinctive terms
• Foundational for many modern keyphrase extraction pipelines

TF-IDF remains a reliable baseline for identifying discriminative terms that differentiate one document from others in a collection.

A query-focused extraction metric that balances relevance against redundancy to produce diverse keyphrase sets.

• Selects phrases that are highly relevant to the document topic
• Penalizes phrases that are too similar to already-selected terms
• Uses cosine similarity between candidate embeddings to measure novelty
• Prevents the extraction of near-duplicate phrases

MMR ensures that extracted keyphrases provide comprehensive topical coverage rather than clustering around a single dominant concept.

The computational process of assigning a numerical weight to each entity or phrase to quantify its contextual importance.

• Goes beyond frequency to consider syntactic position (subject vs. object)
• Incorporates semantic role labeling to identify agent and patient entities
• Uses centering theory to track discourse focus across sentences
• Outputs a ranked list of entities by their prominence in the document

Salience scoring is the bridge between raw keyphrase extraction and entity-aware optimization for AI-driven search engines.