Hierarchical Dirichlet Process (HDP)

Unlike Latent Dirichlet Allocation (LDA), which requires pre-specifying the number of topics K, HDP infers the optimal number of topics directly from the corpus. It uses a Dirichlet Process (DP) as a prior over an infinite topic space, allowing the model to grow in complexity as more data is observed. This eliminates the need for expensive hyperparameter grid searches over K and prevents model misspecification.

HDP formalizes topic sharing across documents using the Chinese Restaurant Franchise (CRF) metaphor:

• Global Level: A Dirichlet Process defines an infinite set of global topics (the 'menu').
• Document Level: Each document is a separate Dirichlet Process whose base measure is the global DP. This hierarchical construction ensures that all documents draw from the same infinite set of topics, but each document exhibits a unique, sparse mixture of those shared themes.

The stick-breaking process provides an explicit representation of the Dirichlet Process prior. A unit-length stick is iteratively broken into pieces, where each piece represents the probability weight of a topic. Formally, the weights β are drawn as:

• β_k = v_k ∏_{i=1}^{k-1} (1 - v_i)
• where v_i ~ Beta(1, γ) The concentration parameter γ controls the rate of decay, determining how many topics receive significant probability mass before the stick is effectively exhausted.

Posterior inference in HDP is typically performed using collapsed Gibbs sampling, a Markov Chain Monte Carlo (MCMC) method. The sampler iteratively resamples the latent topic assignment for each word token, conditioned on all other assignments. Two key sampling steps are involved:

• Sampling the table assignment for each word within a document.
• Sampling the dish (global topic) assigned to each table. This process converges to the true posterior distribution over topic assignments and hyperparameters.

HDP is governed by two critical concentration hyperparameters that control sparsity:

• γ (Gamma): The concentration parameter for the global Dirichlet Process. Lower values of γ favor fewer global topics, enforcing a compact topic space.
• α₀ (Alpha-0): The concentration parameter for the document-level Dirichlet Processes. Lower values of α₀ encourage each document to be composed of fewer distinct topics. Both parameters can be learned from the data by placing vague Gamma priors on them and sampling during inference.

Since HDP automatically determines the number of topics, model quality is assessed using Topic Coherence metrics rather than predictive likelihood alone. Common approaches include:

• C_V Coherence: Measures the semantic interpretability of topics by quantifying co-occurrence of top-ranked words in reference corpora.
• Topic Intrusion: Human evaluation where annotators identify an outlier word injected into a topic's top terms. High coherence scores indicate that the inferred topics are semantically meaningful and interpretable to domain experts.

A stochastic process used in Bayesian nonparametrics whose realizations are probability distributions. It is parameterized by a base distribution (H) and a concentration parameter (α). The DP serves as the foundational prior for the infinite mixture components in HDP, allowing the model to grow its number of topics as data demands. A key property is the Chinese Restaurant Process representation, which provides an intuitive generative story for clustering.

The generative metaphor for the Hierarchical Dirichlet Process. It extends the Chinese Restaurant Process to a franchise setting:

• Restaurants: Represent individual documents.
• Dishes: Represent global topics shared across the entire corpus.
• Tables: Represent clusters of words within a document. Customers (words) sit at tables, and tables are served a dish (topic) from a global menu. This construction naturally enforces topic sharing across documents.

An explicit representation of the Dirichlet Process that defines an infinite mixture of weights. A unit-length stick is iteratively broken:

• β_k: The fraction of the remaining stick broken off at step k.
• Weights (π): The sequence of broken pieces, which sum to 1. In HDP, a global stick-breaking process defines the corpus-wide topic popularities, while document-specific topic proportions are drawn from a DP with this global measure as its base distribution.

A Markov Chain Monte Carlo (MCMC) algorithm used for posterior inference in HDP. It works by iteratively sampling latent topic assignments for each word token:

• Direct Assignment: A collapsed Gibbs sampler that integrates out the infinite topic and document-topic distributions.
• Table Assignment: First samples table assignments for words, then samples dish (topic) assignments for tables. This approach avoids explicitly representing the infinite-dimensional parameter space, making inference computationally tractable.

Key evaluation metrics for HDP models:

• Perplexity: Measures the model's predictive power on held-out data. Lower perplexity indicates better generalization. It is computed as the inverse probability of the test set, normalized by word count.
• Topic Coherence: Quantifies the semantic interpretability of discovered topics. C_V coherence uses normalized pointwise mutual information (NPMI) and cosine similarity over word context vectors to score topics, correlating strongly with human judgment of topic quality.

The parametric predecessor to HDP. LDA requires the number of topics (K) to be fixed a priori, whereas HDP infers K from the data. Both are generative probabilistic models that represent documents as mixtures of topics. HDP generalizes LDA by placing a Dirichlet Process prior on the topic space, allowing for an unbounded number of topics. This makes HDP ideal for exploratory analysis where the true number of themes is unknown.