Latent Dirichlet Allocation (LDA)

LDA models documents as a generative process where each word is created by first selecting a topic from a per-document distribution, then drawing a word from that topic's distribution. This Bayesian framework assumes documents are random mixtures over latent topics, making it a true probabilistic graphical model. The generative story involves two key steps: choosing a topic mixture θ for the document, then for each word position, sampling a topic assignment z and finally the word w.

The model uses Dirichlet distributions as conjugate priors to enforce realistic constraints on topic mixtures. The Alpha hyperparameter controls per-document topic sparsity—lower values force documents to contain fewer topics. The Beta hyperparameter controls per-topic word sparsity—lower values force topics to use fewer, more specific words. This dual-prior structure prevents overfitting and produces more interpretable, coherent topics by penalizing uniform distributions.

LDA treats documents as bag-of-words representations, discarding word order and syntax entirely. This exchangeability assumption means the model considers only word co-occurrence patterns across the corpus. While this simplifies computation, it means LDA cannot capture phrases, negations, or sequential context. The input is typically a Document-Term Matrix (DTM) where rows are documents, columns are vocabulary terms, and values are raw or normalized token frequencies.

Since exact posterior inference is intractable, LDA relies on Gibbs Sampling, a Markov Chain Monte Carlo method. The algorithm iteratively resamples each word token's topic assignment conditioned on all other assignments. Key steps:

• Initialize random topic assignments for every word
• For each token, compute probability of each topic given current state
• Reassign token to sampled topic
• Repeat until convergence of the document-topic and topic-word distributions

LDA requires pre-specifying K—the number of topics—as a fixed hyperparameter. This parametric constraint is both a strength and limitation: it forces explicit model selection but cannot adapt to corpora with unknown thematic breadth. Practitioners typically use perplexity scores on held-out data or topic coherence metrics like C_V to compare models across different K values. The optimal K balances granularity against interpretability and generalization.

Beyond perplexity, modern LDA evaluation relies on topic coherence—measuring semantic interpretability by quantifying word co-occurrence in reference corpora. C_V coherence combines normalized Pointwise Mutual Information (PMI) with cosine similarity over word context vectors, correlating strongly with human judgments. Additional diagnostics include topic diversity (uniqueness of top-N words across topics) and topic intrusion tasks where annotators identify injected outlier words.

A probability distribution over probability vectors used as a conjugate prior in Bayesian topic models. It controls the sparsity of distributions.

• Alpha (α): Controls document-topic sparsity. Low α forces documents to contain fewer topics.
• Beta (β): Controls topic-word sparsity. Low β forces topics to use fewer, more specific words.
• Conjugate Prior: Mathematically convenient because the posterior distribution remains a Dirichlet, simplifying inference.

A Markov Chain Monte Carlo (MCMC) algorithm for approximate inference in LDA. It iteratively samples latent topic assignments for each word token conditioned on all other assignments.

• Collapsed Gibbs Sampling: Integrates out the multinomial parameters, sampling only the latent topic assignments z.
• Process: Randomly initializes topics, then repeatedly resamples each word's topic based on the document's topic mixture and the topic's word distribution.
• Convergence: Monitored via log-likelihood or perplexity stabilization on held-out data.

An optimization-based alternative to Gibbs sampling that finds the closest tractable distribution to the true posterior by minimizing Kullback-Leibler (KL) divergence.

• Mean-Field Assumption: Assumes the variational distribution factorizes over latent variables for tractability.
• Evidence Lower Bound (ELBO): Maximized as a proxy for minimizing KL divergence.
• Advantage: Scales efficiently to large corpora and integrates naturally with stochastic gradient optimization for online learning.

An evaluation metric measuring the semantic interpretability of a topic by quantifying co-occurrence between top-ranked words in reference corpora.

• C_V Coherence: Combines normalized Pointwise Mutual Information (PMI) with cosine similarity over word context vectors. Correlates highly with human judgments.
• UCI Coherence: Based on PMI between word pairs using sliding window counts.
• NPMI: Normalized PMI variant ranging from -1 to 1, reducing sensitivity to word frequency.

A predictive metric measuring how well a topic model generalizes to unseen documents. Calculated as the inverse probability of the test set, normalized by word count.

• Formula: exp(-log likelihood / total word tokens).
• Interpretation: Lower perplexity indicates better generalization. A perplexity of 100 means the model is as confused as if choosing uniformly from 100 words.
• Limitation: Does not correlate with human interpretability. Often used alongside coherence for a complete evaluation picture.

An interactive visualization library for interpreting topic models by projecting inter-topic distances onto a 2D plane using multidimensional scaling.

• Intertopic Distance Map: Shows topic centroids as circles. Circle area indicates topic prevalence. Distance between circles reflects semantic similarity.
• Salient Terms: A relevance metric (λ) balances a term's frequency within a topic against its lift over corpus-wide background frequency.
• Relevance Tuning: Adjusting λ from 0 (most specific terms) to 1 (most probable terms) helps users explore topic-word relationships.