Pointwise Mutual Information (PMI)

The primary evaluation metric that quantifies semantic interpretability of discovered topics. PMI serves as the foundational building block for coherence measures like C_V coherence and UMass coherence. Topic coherence measures the degree of semantic similarity between a topic's top-N words by calculating their co-occurrence statistics in reference corpora.

• Intrinsic measures: Use the original corpus (e.g., UMass)
• Extrinsic measures: Use external corpora like Wikipedia (e.g., C_V, NPMI)
• Higher coherence scores correlate strongly with human interpretability judgments

A state-of-the-art coherence metric that combines normalized PMI with cosine similarity over word context vectors. It segments top-N words into pairs, computes confirmation measures using indirect cosine similarity, and averages them. C_V consistently achieves the highest correlation with human topic interpretability ratings.

• Uses a sliding window to build context vectors for each word
• Computes cosine similarity between context vectors of word pairs
• Applies NPMI as the confirmation measure within the cosine framework

A variant of PMI that constrains values to the range [-1, +1] by dividing raw PMI by the negative log of joint probability. NPMI corrects PMI's bias toward rare words and enables meaningful comparisons across different word pairs.

• -1: Words never co-occur (perfect negative association)
• 0: Words co-occur as expected by chance (independence)
• +1: Words always co-occur (perfect positive association)
• Preferred over raw PMI in modern coherence calculations

A human-in-the-loop evaluation method that directly validates topic interpretability. Annotators are shown a topic's top-N words plus one randomly injected intruder word from another topic. The ease with which humans detect the outlier measures the topic's semantic coherence.

• Topic Intrusion Score: Percentage of correct intruder identifications
• Complements automated metrics like PMI-based coherence
• Gold standard for validating that automated metrics align with human judgment

The generative probabilistic model where PMI-based coherence is most commonly applied. LDA represents documents as random mixtures over latent topics, with each topic characterized by a Dirichlet distribution over words. PMI evaluates the quality of these learned word distributions.

• Alpha: Controls document-topic sparsity
• Beta: Controls topic-word sparsity
• PMI coherence helps select the optimal number of topics (K) by measuring interpretability across different K values

An intrinsic coherence measure that calculates PMI using document co-occurrence counts from the original training corpus rather than an external reference corpus. It defines co-occurrence as two words appearing together in the same document.

• Computationally efficient: no external corpus needed
• Uses smoothed PMI with a small epsilon to handle zero counts
• Often lower correlation with human judgment compared to extrinsic measures like C_V
• Formula: sum over word pairs of log(P(wi,wj)+epsilon / P(wj))