Phrase Candidate Generation

Applies regular expression patterns over POS tag sequences to extract candidates matching specific syntactic structures. The most common pattern is zero or more adjectives followed by one or more nouns (JJ)*(NN)+, which captures descriptive noun phrases like 'deep convolutional neural network' while filtering out verbs and prepositions. More sophisticated systems use chunk grammar rules to define custom patterns for domain-specific terminology extraction.

Uses stopwords and punctuation as phrase boundary delimiters to segment text into candidate sequences. Algorithms like RAKE split text at stopword positions, treating the contiguous content words between them as candidate keyphrases. This approach is computationally efficient and language-agnostic, requiring only a stopword list rather than a full POS tagger, making it suitable for low-resource languages and rapid prototyping.

Generates all contiguous word sequences up to a maximum length (typically n=1 to n=5) as candidate phrases. While exhaustive, this brute-force approach produces many linguistically invalid candidates like 'the of in'. Modern systems combine n-gram enumeration with syntactic filters to prune sequences that cross clause boundaries or contain incompatible POS transitions, dramatically reducing the candidate space before scoring.

Employs shallow parsing to identify non-recursive noun phrases as the primary candidate source. Unlike full syntactic parsing, chunking uses finite-state rules or sequence labeling models to detect base noun phrases without building complete parse trees. This strikes a balance between linguistic precision and computational efficiency, producing candidates that align with how humans naturally select keyphrases from technical documents.

Applies minimum occurrence counts to filter noise from the candidate set before downstream scoring. Rare n-grams appearing only once are often typographical errors or idiosyncratic expressions rather than meaningful keyphrases. Setting a threshold of 2-3 minimum occurrences eliminates approximately 40-60% of spurious candidates while retaining genuine multi-word terms that exhibit stable document frequency patterns.

Augments the candidate pool by identifying semantically related phrases not captured by surface-form extraction. Using sentence transformers, the system encodes the document and retrieves similar phrases from an external knowledge base or the document itself through cosine similarity search. This bridges the gap between present and absent keyphrase extraction, enabling the discovery of conceptual candidates that share semantic space with the source text.

The foundational NLP task of assigning grammatical categories (e.g., noun, verb, adjective) to each token in a sentence. Candidate generation relies heavily on POS patterns to filter for linguistically valid phrases. Common patterns include sequences of zero or more adjectives followed by one or more nouns, which effectively identify noun phrase chunks. Modern POS taggers use fine-tuned Transformer models to achieve over 97% accuracy on standard benchmarks, providing a robust syntactic scaffold for downstream phrase extraction.

A shallow parsing technique that segments text into non-overlapping, flat phrases based on POS tags. Unlike full dependency parsing, chunking is computationally efficient and directly targets the base noun phrases that form the majority of keyphrase candidates. A typical chunking grammar uses regular expressions over POS tags, such as NP: {<DT>?<JJ>*<NN>} to capture determiners, adjectives, and nouns. This process transforms a tagged sequence like 'The/DT quick/JJ brown/JJ fox/NN' into a single chunk: 'The quick brown fox'.

A brute-force method that generates all contiguous word sequences of length 1 to N from a document. While simple, it produces a high volume of noisy candidates that must be aggressively filtered. Common constraints include:

• Stopword boundary filtering: preventing n-grams from starting or ending with stop words
• Frequency thresholds: discarding n-grams appearing fewer than 2-3 times
• Maximum length: typically capping at 5-grams to avoid unwieldy phrases This approach is often combined with POS filtering to improve precision.

A core mechanism in algorithms like RAKE (Rapid Automatic Keyword Extraction) that uses a predefined list of stop words and phrase delimiters to segment text into candidate phrases. The algorithm splits the document at each stopword or punctuation mark, treating the resulting contiguous word sequences as initial candidates. For example, 'Features of phrase candidate generation' yields candidates 'Features', 'phrase candidate generation'. This method is language-agnostic and requires no training data, making it highly portable across domains.

A set of rule-based post-processing steps applied after initial candidate generation to prune invalid or low-quality phrases. Essential heuristics include:

• Minimum occurrence: removing candidates that appear only once
• Nested phrase resolution: if 'neural network' and 'deep neural network' are both candidates, keeping only the longer, more specific form
• Punctuation and digit scrubbing: stripping leading/trailing special characters
• Case normalization: lowercasing all candidates for deduplication while preserving acronyms These rules dramatically reduce the candidate set size before computationally expensive scoring.

The standard metrics for assessing candidate generation quality against a gold-standard set of author-assigned keyphrases. Precision@K measures the fraction of the top-K predicted candidates that are correct, while Recall@K measures the fraction of all gold-standard keyphrases found in the top-K predictions. The F1@K score provides a harmonic mean. For candidate generation specifically, high recall is prioritized—the goal is to ensure the true keyphrases survive the filtering stage, even at the cost of including some noise that downstream scoring will eliminate.