Automatic Indexing

The engine analyzes raw text to compute statistical salience without external knowledge bases. Key metrics include:

• Term Frequency (TF): Raw count of a term in a document.
• Inverse Document Frequency (IDF): Penalizes terms common across a corpus.
• Co-occurrence Strength: Measures how often words appear together in a sliding window. Algorithms like RAKE and YAKE rely solely on these internal document statistics to identify candidate keyphrases, making them domain-independent and computationally lightweight.

Documents are modeled as co-occurrence graphs where nodes represent words or phrases and edges represent semantic or syntactic relationships. The system applies centrality algorithms to identify dominant nodes:

• TextRank: Adapts Google's PageRank to text, iteratively scoring nodes based on the importance of their neighbors.
• SingleRank: Extends TextRank by weighting edges with co-occurrence counts.
• PositionRank: Biases the graph by adding a weight for words appearing early in the document. This method excels at identifying present keyphrases without supervised training data.

Modern systems use transformer models to convert documents and candidate phrases into dense vector representations. Indexing becomes a semantic similarity task:

• KeyBERT: Generates document-level embeddings and selects candidate n-grams with the highest cosine similarity to the document vector.
• EmbedRank: Computes sentence embeddings for each candidate phrase and ranks them by similarity to the document embedding. This approach captures latent semantic meaning, allowing the system to identify phrases that are conceptually central even if they appear infrequently.

Rather than extracting free-text phrases, the system maps document content to a predefined taxonomy or thesaurus. The process involves:

• Entity Linking: Identifying textual mentions and grounding them to unique knowledge base identifiers.
• Wikification: Automatically linking phrases to Wikipedia articles as canonical representations.
• Ontology Alignment: Matching extracted concepts to a structured hierarchy of domain terms. This ensures consistency across indexed collections, eliminating the synonym and homonym problems inherent in free-text keyphrasing.

Before scoring, the system must generate a set of viable phrase candidates. This pipeline typically includes:

• Part-of-Speech (POS) Tagging: Identifying noun phrases and adjective-noun patterns using regular expression chunking like (JJ)*(NN)+.
• Stopword Delimiting: Using closed-class words as phrase boundaries to segment runs of content words.
• N-gram Pruning: Limiting candidates to a maximum length (e.g., trigrams) to reduce noise.
• Phraseness Scoring: Filtering out sequences that are not linguistically well-formed phrases, independent of their topical relevance.

Raw relevance scoring often produces a list of near-duplicate keyphrases. Automatic indexing systems apply Maximal Marginal Relevance (MMR) to diversify results:

• The algorithm iteratively selects the highest-scoring candidate.
• It then penalizes remaining candidates proportional to their similarity to already-selected phrases.
• The balance between relevance and novelty is controlled by a tunable lambda parameter. This ensures the final index terms provide broad topical coverage of the document rather than repeating the same concept.