Latent Dirichlet Allocation (LDA)

The overarching statistical method for discovering abstract themes within a document collection. While LDA is the most famous implementation, topic modeling encompasses a family of algorithms that identify patterns of word co-occurrence to reveal latent semantic structures. Key distinction: LDA assumes documents are mixtures of topics, and topics are mixtures of words. Alternative approaches include Non-Negative Matrix Factorization (NMF) and Latent Semantic Analysis (LSA).

The mathematical foundation that gives LDA its name. A Dirichlet distribution is a probability distribution over probability simplexes—essentially, a distribution of distributions. In LDA, it serves as the conjugate prior for the multinomial distribution, governing how topics are mixed in documents and how words are mixed in topics. The alpha hyperparameter controls document-topic density, while beta controls topic-word density. Higher alpha values produce documents with more mixed topics; lower values yield documents dominated by fewer topics.

A foundational text representation method that often serves as a preprocessing step or baseline comparison for LDA. Term Frequency-Inverse Document Frequency weights words by how often they appear in a document, offset by how common they are across the entire corpus. Unlike LDA's probabilistic generative approach, TF-IDF is a deterministic weighting scheme. It excels at surfacing distinctive terms but cannot capture the polysemy or thematic relationships that LDA uncovers through co-occurrence patterns.

The two primary metrics for evaluating LDA model quality. Perplexity measures how well the model predicts unseen documents—lower scores indicate better generalization, but it correlates poorly with human interpretability. Topic Coherence scores (such as C_v or UMass) measure the semantic similarity between top words in a topic. High coherence indicates human-interpretable topics. Modern best practice: optimize for coherence rather than perplexity when building models for human consumption.

A Markov Chain Monte Carlo (MCMC) algorithm commonly used for approximate inference in LDA. Rather than computing the intractable posterior distribution directly, Gibbs sampling iteratively assigns each word to a topic based on the current assignments of all other words. The process converges to the true posterior after sufficient iterations. Collapsed Gibbs Sampling integrates out the multinomial parameters, sampling only the latent topic assignments, which dramatically reduces computational complexity.

A contrasting approach to text representation where words are mapped to dense vectors in a continuous space. Unlike LDA's discrete topic assignments, embeddings like Word2Vec and GloVe capture semantic relationships through vector proximity. Modern topic modeling often combines these paradigms—using embeddings to initialize or refine LDA topics, or employing neural variational inference in models like Embedded Topic Model (ETM) that marry the interpretability of LDA with the semantic richness of embeddings.