Topic Diversity

The foundational metric for topic diversity is the percentage of unique words across the top-N terms of all discovered topics. A model with high diversity avoids term overlap, where the same high-probability words like 'system' or 'data' dominate multiple topics. The calculation is straightforward: divide the number of unique words in the union of all top-N topic term sets by the total number of possible slots (N × K). A score of 1.0 indicates perfect uniqueness, while a score near 0.0 signals severe topic redundancy.

Rank-Biased Overlap provides a more nuanced measure than simple set intersection by accounting for the rank position of terms. Unlike Jaccard similarity, RBO applies a decaying weight to lower-ranked terms, making it top-weighted and well-suited for evaluating topic model quality. When comparing two topics, an RBO score near 0 indicates high diversity, while scores approaching 1 suggest the topics are near-duplicates. This metric is particularly useful for tuning the alpha hyperparameter to control topic sparsity.

A global diversity score can be derived by calculating the average pairwise similarity between all topic vectors and taking the inverse. Common similarity measures include:

• Cosine similarity on topic-word probability distributions
• Jensen-Shannon divergence between topic distributions
• Spearman's rank correlation on top-N term lists A high average similarity indicates the model has collapsed into a few redundant themes, a common failure mode when the number of topics (K) is set too high relative to the corpus's true thematic granularity.

Effective topic diversity balances two competing objectives: maximizing the semantic breadth covered by the topic set while minimizing inter-topic redundancy. A model with 100 topics that all describe minor variations of 'sports' has high redundancy and low effective coverage. The intertopic distance map from pyLDAvis provides a visual diagnostic—well-dispersed topic centroids indicate healthy diversity, while tight clusters signal that the model has overfit to a dominant theme. This balance is critical for downstream tasks like document clustering and information retrieval.

Topic diversity serves as a critical signal for tuning model hyperparameters. A sudden drop in diversity as K increases often indicates the model has exceeded the corpus's intrinsic dimensionality. Similarly, the beta hyperparameter directly controls term sparsity within topics—lower beta values produce more focused, diverse topics by concentrating probability mass on fewer words. Monitoring diversity alongside topic coherence prevents the common pitfall of optimizing for interpretability at the cost of producing a model where all topics converge on the same high-frequency corpus terms.

Modern neural approaches like BERTopic introduce new diversity considerations. Unlike LDA's explicit Dirichlet priors, BERTopic relies on HDBSCAN clustering of document embeddings followed by class-based TF-IDF. Diversity in this context is influenced by the clustering algorithm's minimum cluster size and the dimensionality reduction technique (UMAP). A model with high diversity but low topic coherence may indicate that the clustering has fragmented genuine themes into artificial micro-clusters, requiring adjustment of the clustering sensitivity parameters.

The primary metric for evaluating the semantic interpretability of a topic. It quantifies the degree of co-occurrence between a topic's top-ranked words in a reference corpus.

• C_V Coherence: The most robust variant, combining normalized pointwise mutual information (NPMI) with cosine similarity over word context vectors. It correlates highly with human judgment.
• U_Mass Coherence: A symmetric measure based on document co-occurrence counts within the training corpus itself, without external reference.
• High coherence scores are the ultimate goal, confirming that a topic's words form a semantically consistent theme rather than a random collection.

An information-theoretic measure of association between two words, forming the mathematical foundation of many topic coherence metrics.

• Formula: PMI(w1, w2) = log(P(w1, w2) / (P(w1) * P(w2))). It measures how much more often two words co-occur than if they were independent.
• Normalized PMI (NPMI): Scales PMI to the range [-1, 1], where -1 means words never co-occur, 0 means independence, and +1 means perfect co-occurrence.
• NPMI is the core building block for calculating C_V coherence, ensuring that a topic's top terms are not just frequent but genuinely associated.

A predictive metric that measures how well a probabilistic topic model generalizes to unseen documents. It calculates the inverse probability of a held-out test set, normalized by the total number of words.

• Lower is Better: A lower perplexity score indicates the model is less 'surprised' by new data, suggesting better generalization.
• Limitations: Perplexity does not always correlate with human interpretability. A model with excellent perplexity can still produce incoherent topics, which is why it's often paired with coherence measures.
• It is mathematically defined as exp(- (sum of log probabilities) / total word count).

A human-in-the-loop evaluation method that directly measures the interpretability of a topic's latent space. Annotators are presented with the top-N words of a topic plus one randomly injected outlier word.

• The Task: The human must identify the intruder word that does not belong to the semantic theme.
• Intrusion Score: A high detection rate means the topic is highly interpretable and semantically tight. If annotators frequently fail, the topic's word distribution is noisy.
• This method provides a ground-truth check on automated metrics like coherence, ensuring models produce themes that make intuitive sense to humans.

Prior concentration parameters in Latent Dirichlet Allocation (LDA) that directly control the sparsity of distributions, heavily influencing topic diversity.

• Alpha (α): Controls the per-document topic distribution. A low α enforces that each document is composed of only a few dominant topics.
• Beta (β): Controls the per-topic word distribution. A low β enforces that each topic is composed of a small set of very specific, highly probable words.
• Tuning for Diversity: Setting a low β is a direct way to increase topic diversity by forcing topics to focus on fewer, more distinct terms, reducing overlap between topics.