Number of Topics (K)

Unlike nonparametric models like the Hierarchical Dirichlet Process, setting K imposes a hard constraint on the number of latent themes. The model must compress the entire semantic diversity of the corpus into exactly K distinct topic buckets. This requires the practitioner to possess strong prior knowledge or to rely on quantitative metrics to select the optimal value before inference begins.

K directly controls the resolution of the semantic lens:

• Low K (e.g., 5-10): Produces broad, high-level themes (e.g., 'Sports', 'Politics'). High generalization but risks merging distinct sub-themes.
• High K (e.g., 100-500): Produces fine-grained, specific themes (e.g., 'Free Agency Negotiations', 'City Council Zoning Laws'). High specificity but risks fragmenting coherent themes and creating 'junk' topics.

K exhibits a classic bias-variance trade-off visible in evaluation metrics:

• Perplexity Score: Often improves (decreases) monotonically with higher K because the model fits the training data better. However, this leads to overfitting and poor generalization if K is too high.
• Topic Coherence (C_V): Typically peaks at an optimal K and then declines. A K that is too high creates topics dominated by idiosyncratic word co-occurrences rather than semantically meaningful patterns.

K is a primary multiplier of computational cost. The time complexity of Gibbs Sampling for LDA scales linearly with K. Doubling the number of topics roughly doubles the memory required to store the Document-Topic Distribution matrix and the time required for each iteration of inference. In production environments, K must be balanced against latency budgets.

K does not act in isolation; it interacts critically with the Alpha Hyperparameter. For a fixed Alpha, increasing K sparsifies the per-document topic distribution, as the probability mass must be divided among more topics. To maintain dense topic mixtures with a high K, Alpha must be increased proportionally to counteract the dimensionality effect.

Quantitative metrics alone are insufficient to select K. The final selection must be validated through qualitative inspection using tools like pyLDAvis and Topic Intrusion tests. A model with a slightly lower coherence score but highly distinct, interpretable topics on the Intertopic Distance Map is often superior to a mathematically optimal but semantically noisy model.

A predictive metric measuring how well a topic model generalizes to unseen documents by calculating the inverse probability of the test set, normalized by word count. Lower perplexity indicates better predictive performance.

• Limitation: Does not correlate with human interpretability
• Calculation: exponential of the negative log-likelihood per word
• Use Case: Monitoring convergence during training

An evaluation metric that measures the semantic interpretability of a topic by quantifying the degree of co-occurrence between its top-ranked words in reference corpora. High coherence scores indicate that top words logically belong together.

• C_V Coherence: Combines normalized pointwise mutual information with cosine similarity over word context vectors
• UCI Coherence: Based on pointwise mutual information over sliding windows
• NPMI: Normalized variant robust to corpus size

A metric assessing the uniqueness of topics by calculating the percentage of unique words across the top-N terms of all discovered topics. High diversity prevents topic redundancy.

• Formula: (Unique words in top-N across all topics) / (N × K)
• Trade-off: Excessive K inflates diversity but fragments semantics
• Target: Balance diversity with coherence for optimal K

An interactive visualization library for interpreting topic models by projecting inter-topic distances onto a two-dimensional plane using multidimensional scaling. Essential for qualitative K selection.

• Intertopic Distance Map: Visualizes topic overlap and distinctness
• Relevance Metric: Balances term frequency within a topic against corpus-wide lift
• Workflow: Adjust K, re-run model, and visually inspect cluster separation

An evaluation method where human annotators identify an injected outlier word within a topic's top terms. Measures the interpretability of the latent space directly from human judgment.

• Procedure: Present top-5 words plus one random low-probability word
• Metric: Topic intrusion score = fraction of correctly identified intruders
• Correlation: High intrusion scores align with high C_V coherence

A nonparametric Bayesian model that infers the number of topics from data by placing a Dirichlet process prior on the topic space, allowing for infinite topic mixtures. Eliminates the need to manually set K.

• Mechanism: Uses a stick-breaking process to grow topics as data demands
• Advantage: K adapts to corpus complexity automatically
• Trade-off: Computationally more expensive than fixed-K LDA