Topic Modeling

A generative probabilistic model that assumes each document is a mixture of a small number of topics, and each word's occurrence is attributable to one of the document's topics.

• Dirichlet priors control the sparsity of topic distributions per document and word distributions per topic
• Uses Gibbs sampling or variational inference to estimate posterior distributions
• Outputs two matrices: document-topic probabilities and topic-word probabilities
• Best suited for moderate-sized corpora where topics are semantically distinct

Example: A 10,000-document news corpus might yield topics like 'sports', 'politics', and 'technology', with each article assigned a percentage breakdown across these themes.

A linear algebra approach that factorizes the document-term matrix into two lower-rank non-negative matrices representing topics and their weights.

• Enforces non-negativity constraints, producing interpretable, additive topic representations
• Uses multiplicative update rules or coordinate descent for optimization
• Tends to produce more coherent, localized topics than LDA on shorter texts
• Requires preprocessing like TF-IDF vectorization; works well with Frobenius norm or Kullback-Leibler divergence objectives

Example: Applied to customer support tickets, NMF can surface distinct issue clusters like 'login failure', 'billing dispute', and 'shipping delay' without probabilistic assumptions.

An early vector space method that applies Singular Value Decomposition (SVD) to the term-document matrix to reduce dimensionality and reveal latent semantic relationships.

• Captures synonymy and polysemy by mapping terms and documents into a lower-dimensional semantic space
• Truncated SVD retains only the top-k singular values, discarding noise
• Lacks probabilistic grounding; topics can contain negative weights, reducing interpretability
• Serves as a foundational precursor to modern embedding-based approaches

Example: LSA can identify that 'car' and 'automobile' co-occur in similar contexts even when they never appear in the same document, enabling conceptual retrieval beyond exact keyword matching.

A modern neural approach that leverages transformer-based embeddings and class-based TF-IDF to create dense, coherent topic clusters.

• Uses sentence transformers to generate contextual document embeddings, capturing semantic nuance beyond bag-of-words
• Applies HDBSCAN for density-based clustering, automatically identifying outliers
• Employs c-TF-IDF to weight terms within clusters rather than across the full corpus
• Supports dynamic topic modeling to track how topics evolve over time

Example: Analyzing 50,000 product reviews, BERTopic can distinguish between 'battery life complaints', 'screen quality praise', and 'customer service frustration' with high granularity, even when vocabulary overlaps.

An extension of LDA that replaces the Dirichlet prior with a logistic normal distribution, allowing topics to exhibit correlation rather than near-independence.

• Models topic co-occurrence: a document about 'genetics' is more likely to also discuss 'bioinformatics' than 'real estate'
• Uses logistic normal distribution to capture covariance structure between topics
• Estimated via variational EM with non-conjugate inference
• Provides a more realistic document representation for corpora where themes naturally overlap

Example: In a corpus of academic papers, CTM reveals that 'machine learning' and 'statistics' topics are positively correlated, while 'machine learning' and 'medieval literature' are not, enabling richer content recommendations.

A generalized framework that incorporates document-level metadata or covariates directly into the topic modeling process, enabling analysis of how topical prevalence and content vary with external factors.

• Allows covariates (e.g., publication date, author, region) to influence topic prevalence and word choice within topics
• Supports interaction effects between covariates
• Built on a variational EM estimation strategy
• Widely used in social science research to study how discourse changes across conditions

Example: Analyzing political speeches, STM can reveal that the prevalence of the 'climate change' topic increases over time and differs systematically by party affiliation, while the vocabulary used to discuss it also shifts.

The foundational generative probabilistic model that views documents as a mixture of topics and topics as a mixture of words. LDA uses Dirichlet priors to assume a document contains multiple topics in varying proportions.

• Iterates through each word, assigning it to a topic based on document-topic and topic-word distributions
• Outputs two matrices: document-topic probabilities and topic-word probabilities
• Hyperparameters alpha and beta control topic sparsity and word distribution smoothness
• Commonly implemented via Gibbs sampling or variational inference

A numerical statistic that converts raw text into machine-readable vectors by weighing term frequency against inverse document frequency. It penalizes common words while rewarding distinctive terms.

• Term Frequency (TF): How often a word appears in a document
• Inverse Document Frequency (IDF): Log-scaled ratio of total documents to documents containing the term
• Serves as the input matrix for many topic modeling algorithms
• Largely superseded by word embeddings for semantic tasks but remains critical for keyword-based topic discovery

A dense vector representation where semantically similar words map to nearby points in high-dimensional space. Unlike sparse TF-IDF vectors, embeddings capture contextual relationships and analogies.

• Word2Vec uses skip-gram or CBOW architectures to learn from co-occurrence
• GloVe factorizes a word-word co-occurrence matrix
• Enables topic modeling via clustering: k-means on averaged word vectors reveals thematic groupings
• Modern contextual embeddings like BERT produce dynamic representations that change based on surrounding words

A distance metric that quantifies how closely two documents or terms align in meaning. Essential for evaluating topic coherence and grouping related content.

• Cosine similarity measures the angle between document vectors, ranging from -1 to 1
• Topic models are validated by checking if top words in a topic are semantically related
• Topic coherence metrics like C_V and NPMI use semantic similarity to score topic quality
• Powers content recommendation engines by matching documents with similar topic distributions

The downstream application of topic modeling where documents are automatically assigned to predefined categories. Topic distributions serve as feature vectors for supervised classifiers.

• Multi-label classification handles documents that span multiple topics
• Topic model outputs reduce dimensionality: a 100-topic vector replaces a 50,000-word vocabulary
• Common classifiers: Support Vector Machines, Logistic Regression, and Random Forests
• Enables automated metadata tagging pipelines that assign editorial categories at scale

The task of identifying salient terms that best summarize a document's core subjects. Complements topic modeling by surfacing specific terminology rather than abstract themes.

• Unsupervised methods like TextRank build word graphs and apply PageRank to score importance
• Supervised approaches treat extraction as a binary classification of candidate phrases
• Topic modeling provides the global context; keyphrase extraction provides the local specifics
• Together they power automated metadata generation for SEO titles and meta descriptions