Topic Intrusion

The core mechanism involves presenting an annotator with a set of top-N terms from a topic, plus one intruder word randomly selected from a high-probability term in a different, unrelated topic. The annotator's job is to identify the word that does not belong. High model precision is indicated when the intruder is easily spotted, confirming the topic's internal semantic coherence.

The final metric is the Topic Intrusion Score, calculated as the fraction of topics where the intruder was correctly identified by the annotator.

• Score = (Number of correct identifications) / (Total number of topics evaluated)
• A score of 1.0 indicates perfect semantic interpretability.
• This provides a direct, human-grounded complement to automated metrics like C_V Coherence.

Topic intrusion is often confused with the related Word Intrusion task, but they evaluate different levels of the model:

• Word Intrusion: Injects an outlier word into a single topic's term list to test the internal consistency of that specific topic.
• Topic Intrusion: Injects an outlier topic into a document's topic distribution to test the semantic validity of the document-topic assignment. Both are standard protocols introduced by Chang et al. (2009) for reading tea leaves.

Unlike purely statistical metrics like Perplexity Score, topic intrusion relies on human judgment as the ground truth. This makes it a critical tool for validating models intended for human consumption, such as exploratory document analysis. The task is typically administered via crowdsourcing platforms, and inter-annotator agreement is measured to ensure the reliability of the intrusion signal.

Topic intrusion is the behavioral counterpart to the mathematical Topic Coherence family of metrics. While Pointwise Mutual Information (PMI) and C_V Coherence score topics based on word co-occurrence in a reference corpus, topic intrusion validates whether those mathematical clusters actually align with human semantic understanding. A high coherence score that fails an intrusion test suggests a disconnect between statistical optimization and interpretability.

Topic intrusion is used as a decisive factor in hyperparameter tuning, specifically for selecting the Number of Topics (K). A model with too many topics often produces fragmented, overlapping themes where intruders are difficult to detect. By plotting the intrusion score against K, practitioners can identify the 'elbow' where adding more topics degrades human interpretability, even if perplexity continues to improve.

An automated evaluation metric that quantifies the semantic interpretability of a topic by measuring the degree of co-occurrence between its top-ranked words in a reference corpus.

• C_V Coherence: Combines normalized pointwise mutual information with cosine similarity over word context vectors.
• UCI Coherence: Based on pointwise mutual information using sliding windows.
• UMass Coherence: Uses document co-occurrence counts without an external corpus.

High coherence scores correlate strongly with human judgments of topic quality.

An information-theoretic measure of association between two words, foundational to calculating topic coherence scores.

• Quantifies how much more frequently two words co-occur than expected by chance.
• Normalized PMI (NPMI) scales values to [-1, 1], where 1 indicates perfect co-occurrence.
• Used as the building block for C_V coherence calculations.

PMI forms the mathematical backbone for validating whether a topic's top terms genuinely belong together semantically.

A predictive metric measuring how well a topic model generalizes to unseen documents by calculating the inverse probability of the test set, normalized by word count.

• Lower perplexity indicates better generalization performance.
• Computed as the exponential of the negative log-likelihood per word.
• Limitation: Perplexity does not always correlate with human interpretability, which is why topic intrusion and coherence are preferred for qualitative evaluation.

Often used alongside intrusion tasks for a complete evaluation picture.

A metric assessing the uniqueness of topics by calculating the percentage of unique words across the top-N terms of all discovered topics in a model.

• High diversity indicates distinct, non-redundant topics.
• Low diversity suggests topic overlap or model degeneration.
• Complements topic intrusion by ensuring topics are not only interpretable but also meaningfully distinct from one another.

Critical for validating that the latent space captures the full breadth of thematic content in a corpus.

A robust topic coherence measure that combines normalized pointwise mutual information with cosine similarity over word context vectors, correlating highly with human interpretability judgments.

• Constructs a context vector for each top word using co-occurrence with all other top words.
• Computes cosine similarity between context vectors.
• Aggregates similarities into a single coherence score per topic.

C_V is the most widely adopted coherence metric because it most accurately mirrors human evaluations like topic intrusion.