KP20k

The dataset aggregates over 20,000 abstracts from a diverse range of computer science domains. This volume provides sufficient training data for complex sequence-to-sequence models and large language model fine-tuning, moving beyond small-scale toy datasets.

• Source: Papers from ACM, ACL, and related digital libraries.
• Scale: 20k+ documents for training, with dedicated validation and test splits.

Unlike datasets relying on crowd-sourced or algorithmic labels, KP20k uses author-assigned keyphrases. This ensures the labels represent the true semantic intent of the paper, capturing both present keyphrases (verbatim in text) and absent keyphrases (inferred concepts).

• Quality: High precision labels reflecting expert domain knowledge.
• Dual Task: Tests both extraction and abstractive generation capabilities.

KP20k evaluates models on two distinct tasks. Present Keyphrase Extraction identifies spans in the source text, while Absent Keyphrase Generation requires the model to synthesize phrases not explicitly written. This dual setup benchmarks both discriminative and generative AI capabilities.

• Metrics: Standard evaluation uses F1@K (often K=5, 10) and Mean Reciprocal Rank (MRR).
• Challenge: Models must balance copying mechanisms with semantic generation.

The standard preprocessing pipeline involves lowercasing, digit replacement with <digit>, and subword tokenization. Text is often truncated to the first 200-400 tokens of the abstract to fit model input limits. Keyphrases are stemmed or lemmatized during evaluation to handle morphological variants.

• Input: Title concatenated with the abstract body.
• Output: A delimited sequence of keyphrases (e.g., separated by ;).

A single source abstract maps to multiple target keyphrases (typically 5-6 on average). This forces models to learn a one-to-many generation paradigm, capturing the multi-faceted nature of scientific documents. It tests the model's ability to avoid redundancy while maximizing informativeness.

• Diversity: Requires coverage of different subtopics within the abstract.
• Ordering: Target keyphrase sequences are often ordered by importance or appearance.

While trained on computer science papers, KP20k models are frequently tested on out-of-domain datasets like Inspec (abstracts) or PubMed (biomedicine). This evaluates the zero-shot domain transfer capability of the extraction model, a critical factor for enterprise deployment where labeled data is scarce.

• Baseline: Tests the robustness of learned semantic patterns.
• Adaptation: Often used as a pre-training step before fine-tuning on proprietary corpora.

KP20k evaluates both extraction paradigms. Present keyphrase extraction identifies phrases that appear verbatim in the source text, while absent keyphrase extraction generates relevant phrases not explicitly stated.

• Present: Sequence labeling or ranking of candidate n-grams
• Absent: Requires generative models (e.g., Seq2Seq) to synthesize novel terms
• KP20k's author-assigned keyphrases include both types, making it a comprehensive benchmark

The dominant paradigm for KP20k is keyphrase generation, where a sequence-to-sequence model learns to map a source document title and abstract directly to a set of keyphrases.

• Copy mechanisms allow models to extract present phrases while generating absent ones
• One2Set architectures predict the entire keyphrase set in parallel, avoiding order bias
• CatSeq and Transformer-based models are common baselines evaluated on KP20k

KP20k performance is measured using F1@K, the harmonic mean of precision and recall calculated over the top-K predicted keyphrases.

• F1@5 and F1@10 are standard cutoff values
• Mean Reciprocal Rank (MRR) evaluates ranking quality by rewarding early correct predictions
• Exact match against author-assigned gold standards is strict; partial matching via stemmed or lemma forms is also reported

Before scoring, systems must generate a pool of candidate phrases. Phraseness measures how linguistically well-formed a candidate is, independent of its topical relevance.

• POS tagging filters for noun phrases (e.g., JJ+NN patterns)
• Stopword-delimited sequences are used by RAKE and YAKE
• N-gram enumeration up to a maximum length (typically 1-5 grams)
• Informativeness scoring then ranks candidates by domain specificity

KP20k serves as a benchmark to compare supervised models against unsupervised baselines. TextRank builds a word co-occurrence graph and applies PageRank, while YAKE uses statistical features from a single document.

• TextRank: Graph-based, requires no training data
• YAKE: Lightweight, relies on casing, position, and frequency features
• Both underperform supervised models on KP20k but provide strong zero-shot baselines

Combining multiple extraction algorithms often yields more robust results. Reciprocal Rank Fusion (RRF) merges ranked lists by summing the reciprocal of each candidate's rank position.

• Reduces variance from any single model's bias
• Effective for combining sparse (TF-IDF) and dense (embedding) scoring methods
• KP20k studies show ensemble methods consistently improve F1@K over individual models