Present Keyphrase Extraction

The defining constraint of present extraction is that selected keyphrases must exist as contiguous n-gram sequences in the source text. This distinguishes it from absent keyphrase generation, which synthesizes novel phrases. Algorithms operate by first generating candidate spans—typically noun phrases identified via Part-of-Speech (POS) tagging and noun phrase chunking—then scoring them for salience. The boundary detection problem requires precise identification of where a multi-word phrase starts and ends, often using BIO tagging (Beginning, Inside, Outside) in sequence labeling frameworks.

The extraction process begins with phrase candidate generation, which produces a set of potential n-grams from the document. Common techniques include:

• POS filtering: Extracting sequences matching patterns like (JJ)*(NN)+ (adjectives followed by nouns)
• Stopword-delimited splitting: Using stopwords as phrase boundaries, as employed by RAKE
• N-gram enumeration: Generating all n-grams up to a maximum length (typically 1-5 grams)
• Noun phrase chunking: Using shallow parsing to identify base noun phrases

Effective candidate generation balances recall (capturing all true keyphrases) against precision (avoiding an unmanageable number of candidates).

Candidate phrases are evaluated along two orthogonal dimensions:

• Phraseness: Measures how linguistically well-formed a candidate is as a phrase, independent of topical relevance. This penalizes fragments like 'the quick' while favoring complete noun phrases like 'quick brown fox'.
• Informativeness: Quantifies how well a candidate captures the document's core topical content. This is typically measured through TF-IDF, TF-ICF (domain specificity), or semantic similarity between the phrase embedding and the document embedding.

Algorithms like YAKE combine these dimensions into a unified score, while others apply them sequentially.

Methods like TextRank and SingleRank construct a co-occurrence graph where vertices represent candidate words or phrases and edges are weighted by co-occurrence within a sliding window. The PageRank algorithm is then applied iteratively:

• Each vertex's score is computed as a weighted sum of its neighbors' scores
• Convergence produces a stationary distribution where highly connected nodes receive higher salience scores
• Post-processing selects top-ranked phrases and may apply Maximal Marginal Relevance (MMR) to reduce redundancy

This unsupervised approach requires no training data and is domain-independent, making it widely applicable.

Modern approaches like KeyBERT and EmbedRank leverage pre-trained transformer embeddings to capture semantic similarity:

• The document embedding is computed by pooling BERT token embeddings (typically using the [CLS] token or mean pooling)
• Each candidate phrase is embedded similarly
• Cosine similarity between phrase and document embeddings serves as the salience score
• KeyBERT enhances this with Max Sum Similarity and Maximal Marginal Relevance variants to balance relevance with diversity

This approach excels at identifying keyphrases that are semantically central even when they don't exhibit high term frequency.

When labeled training data is available (e.g., KP20k, Inspec, SemEval datasets), present extraction is framed as a token-level classification task:

• Each token receives a BIO tag: B-KP (beginning of keyphrase), I-KP (inside keyphrase), or O (outside)
• Architectures like BiLSTM-CRF or fine-tuned BERT models predict these tags
• The CRF layer enforces valid tag transitions (e.g., I-KP must follow B-KP)
• Evaluation uses F1@K with exact span matching against gold-standard annotations

Supervised methods typically outperform unsupervised approaches when sufficient in-domain training data exists.

The initial filtering step that produces a set of potential n-grams from raw text. Part-of-speech (POS) tagging identifies noun phrases using patterns like (JJ)*(NN)+, while stopword filtering delimits boundaries. Common methods include:

• Noun phrase chunking via regular expressions over POS tags
• N-gram enumeration up to a maximum length (typically 5-grams)
• Stopword-based splitting used by RAKE to form candidate sequences Effective candidate generation reduces the search space by 90%+ before scoring begins.

A family of unsupervised algorithms that model text units as vertices in a co-occurrence graph. Edges connect words or phrases appearing within a sliding window. TextRank applies the PageRank formula iteratively until convergence:

• Each node's score = (1-d) + d * Σ(weight_ji * score_j / outdegree_j)
• d is the damping factor, typically 0.85
• Final keyphrases are formed by combining adjacent high-scoring words This approach requires no training data and adapts to any domain.

Modern methods leverage dense vector representations to measure semantic similarity between candidate phrases and the document. KeyBERT computes the cosine similarity between each candidate's embedding and the document-level embedding:

• Document embedding: mean pooling of all token vectors
• Candidate embedding: mean pooling of candidate token vectors
• EmbedRank extends this by using sentence embeddings for richer context
• Maximal Marginal Relevance (MMR) re-ranks to maximize relevance while minimizing redundancy with already-selected phrases

Lightweight, unsupervised algorithms that score candidates using statistical properties of the text. YAKE computes five features per candidate:

• Casing: ratio of capitalized words
• Position: normalized position of first occurrence
• Frequency: term frequency within the document
• Relatedness: co-occurrence with other candidates
• Sentence frequency: how many sentences contain the candidate RAKE scores by the ratio of degree to frequency in the word co-occurrence graph, favoring longer phrases with rare words.

Standard measures for comparing predicted keyphrases against a gold-standard set. F1@K computes the harmonic mean of precision and recall for the top-K predictions:

• Precision@K: correct predictions in top-K / K
• Recall@K: correct predictions in top-K / total gold keyphrases
• Mean Reciprocal Rank (MRR): averages 1/rank of the first correct prediction
• Binary F1 treats extraction as a classification task over the full candidate set These metrics are computed against author-assigned keyphrases in benchmarks like KP20k.

Techniques that combine multiple extraction algorithms to improve robustness. Reciprocal Rank Fusion (RRF) merges ranked lists without score normalization:

• RRF_score(d) = Σ 1/(k + rank_i(d)) across all rankers
• k is a constant (typically 60) to dampen the impact of high ranks
• Ensemble scoring averages normalized scores from TF-IDF, TextRank, and embedding methods
• Weighted voting assigns confidence weights to each algorithm based on historical F1 performance on validation sets