Topic Coherence

The gold standard coherence measure that combines normalized pointwise mutual information (NPMI) with cosine similarity over word context vectors. It operates by creating a sliding window over the reference corpus, constructing a context vector for each top word, and then computing the cosine similarity between a topic's word vectors and their NPMI scores. C_V consistently achieves the highest correlation with human interpretability judgments across multiple studies, making it the recommended default for topic model evaluation.

• Range: 0 to 1, with higher values indicating more coherent topics
• Reference corpus: Typically Wikipedia or the original training corpus
• Implementation: Available in Gensim and Palmetto

A coherence measure based on pointwise mutual information (PMI) between all pairs of top-N words in a topic, using document co-occurrence counts rather than a sliding window. It calculates the PMI for every word pair and averages the results. UCI coherence penalizes word pairs that appear together less frequently than expected by chance, providing a measure grounded in information theory. However, it correlates less strongly with human judgments compared to C_V.

• Formula: Average PMI(w_i, w_j) over all top-N word pairs
• Smoothing: Uses a small epsilon to handle zero co-occurrences
• Sensitivity: More affected by corpus size than window-based methods

An intrinsic coherence measure that calculates the log conditional probability of word co-occurrence using the original training corpus rather than an external reference. It measures how often top topic words co-occur within the same documents, making it computationally efficient since no external corpus is required. UMass coherence is asymmetric—it scores the confirmation of a word given a preceding word—and typically returns negative values, where scores closer to zero indicate higher coherence.

• Formula: log P(w_i, w_j) / P(w_j) with smoothing
• Advantage: No external corpus dependency
• Limitation: Can overfit to the training data's idiosyncrasies

Normalized Pointwise Mutual Information normalizes PMI scores to the range [-1, 1], where -1 indicates words never co-occur, 0 indicates independence, and +1 indicates complete co-occurrence. This normalization enables cross-model comparisons that are not possible with raw PMI. NPMI serves as the foundational building block for C_V coherence but can also be used directly as a standalone measure when aggregated over top-N word pairs.

• Normalization: NPMI = PMI / -log P(w_i, w_j)
• Interpretability: Scores map to intuitive semantic relationships
• Use case: Comparing coherence across different topic models and datasets

A distributional semantics approach that measures topic coherence using pre-trained word embeddings rather than co-occurrence counts. It computes the average pairwise cosine similarity between the embedding vectors of a topic's top-N words. This method captures semantic similarity beyond exact co-occurrence, recognizing that synonyms and related concepts may not appear together in the same sliding window but share similar vector representations.

• Embedding source: Typically Word2Vec or GloVe trained on large corpora
• Advantage: Captures semantic relationships missed by co-occurrence
• Limitation: Sensitive to the quality and domain of the embedding model

A human-in-the-loop evaluation method where annotators are shown the top-N words of a topic plus one injected outlier word from a different topic. The annotator's task is to identify the intruder. A topic model is considered more interpretable if human judges can reliably detect the out-of-place word. Topic intrusion accuracy serves as a direct measure of topic coherence that bypasses proxy metrics entirely.

• Process: Randomly select outlier from low-probability word in another topic
• Metric: Percentage of correct intruder identifications
• Correlation: Strongly aligns with perceived model quality in user studies

The gold standard for topic coherence measurement. It calculates coherence by using a sliding window over a reference corpus to create context vectors for top words, then computes normalized pointwise mutual information (NPMI) and cosine similarity between these vectors. C_V strongly correlates with human judgments of interpretability, making it the preferred metric in tools like Gensim and Palmetto.

A foundational information-theoretic measure of word association. PMI quantifies how much more often two words co-occur than expected by chance:

• High PMI: Words appear together frequently (e.g., 'neural' and 'network')
• Low PMI: Words co-occur randomly
• Normalized PMI (NPMI): Bounds values to [-1, 1], enabling comparison across different topic models and corpora

A human-in-the-loop evaluation method where annotators are shown the top-N words of a topic plus one injected outlier from another topic. The ease with which humans identify the intruder measures topic interpretability. High intrusion detection accuracy indicates semantically coherent topics; low accuracy suggests overlapping or noisy latent spaces.

A predictive metric that evaluates how well a topic model generalizes to unseen documents. Calculated as the inverse probability of a held-out test set normalized by word count:

• Lower perplexity = better predictive performance
• Limitation: Perplexity often negatively correlates with human interpretability, so it should be used alongside coherence measures rather than as a sole evaluation criterion

Measures the uniqueness of discovered topics by calculating the percentage of unique words across the top-N terms of all topics. High diversity indicates distinct, non-redundant topics:

• 0% diversity: All topics share identical top words
• 100% diversity: No overlap in top words across topics
• Often reported alongside coherence to ensure models produce both interpretable and distinct themes

An interactive visualization library for interpreting topic models. It projects inter-topic distances onto a 2D plane using multidimensional scaling and displays the salient terms for each topic. The relevance metric balances a term's frequency within a topic against its lift over corpus-wide background frequency, allowing analysts to visually assess topic overlap and semantic distinctness.