BERTopic

The pipeline begins by converting documents into dense vector representations using pre-trained transformer models. By default, BERTopic uses sentence-transformers to generate semantically rich embeddings that capture contextual meaning beyond simple keyword frequencies.

• Supports any Hugging Face embedding model
• Can integrate spaCy, Gensim, or Word2Vec models
• Enables multilingual topic modeling with cross-lingual embeddings
• Preserves semantic similarity: documents with similar meaning cluster together even with different vocabulary

High-dimensional embeddings are reduced to a lower-dimensional space to combat the curse of dimensionality. BERTopic defaults to UMAP (Uniform Manifold Approximation and Projection), which preserves both local and global data structure better than traditional PCA.

• Default UMAP preserves neighborhood structure for clustering
• Optional PCA or t-SNE for specific use cases
• Handles sparse high-dimensional data efficiently
• Critical step: poor reduction leads to fragmented or merged topics

Reduced embeddings are clustered using HDBSCAN, a density-based algorithm that automatically identifies clusters of varying densities and marks outliers as noise. Unlike K-Means, HDBSCAN does not require pre-specifying the number of topics.

• Noise points are treated as outliers, not forced into topics
• Automatically discovers the natural number of topics
• Handles clusters of different shapes and densities
• Outlier reduction improves topic coherence by excluding ambiguous documents

Instead of calculating term importance across individual documents, BERTopic computes class-based TF-IDF where each topic cluster is treated as a single aggregated document. This identifies words that are frequent within a topic but rare across all topics.

• Formula: weights term frequency by inverse class frequency
• Produces highly discriminative topic-word distributions
• Enables dynamic topic labeling from top-ranked terms
• Outperforms standard TF-IDF for topic representation by modeling inter-topic distinctiveness

BERTopic offers multiple strategies for generating human-readable topic labels beyond raw c-TF-IDF keywords. The Maximal Marginal Relevance (MMR) algorithm balances term relevance with diversity to avoid redundant keywords.

• KeyBERTInspired: uses embedding similarity to refine keywords
• Part-of-Speech filtering: retains only nouns and adjectives
• GPT-based labeling: generates natural language topic names via LLMs
• Zero-shot topic merging: consolidates similar topics using semantic similarity thresholds

BERTopic extends static topic modeling to temporal analysis by tracking how topics evolve over time. Documents are binned by timestamp, and topic frequencies are calculated per time slice to model topic evolution without retraining separate models.

• Topics over Time: visualizes topic frequency trends
• Merging and splitting detection: identifies topic drift
• Normalizes for varying document volumes across time periods
• Enables longitudinal analysis of thematic shifts in document streams

The generative probabilistic model that serves as the statistical foundation for topic modeling. LDA represents documents as random mixtures over latent topics, where each topic is characterized by a distribution over words.

• Uses Dirichlet priors to control sparsity of topic distributions
• Relies on Gibbs sampling or variational inference for posterior approximation
• Requires pre-specifying the number of topics (K) hyperparameter
• Outputs document-topic distributions and topic-word distributions

The standard interpretability metric for assessing topic quality by measuring semantic similarity between top-ranked words. Unlike perplexity scores which optimize for predictive likelihood, coherence correlates with human judgment.

• C_V coherence combines normalized PMI with cosine similarity over word context vectors
• Pointwise Mutual Information (PMI) measures word association strength
• Topic intrusion tasks inject outlier words to test human interpretability
• Topic diversity calculates the percentage of unique words across all topics

A linear algebra alternative to probabilistic topic models that decomposes the document-term matrix (DTM) into two lower-dimensional non-negative matrices. The non-negativity constraint produces additive, parts-based representations that are often more interpretable than LDA for short texts.

• Works directly on TF-IDF weighted matrices
• No distributional assumptions required
• Often combined with BERTopic for class-based TF-IDF refinement

A nonparametric Bayesian model that eliminates the need to pre-specify the number of topics. HDP places a Dirichlet process prior on the topic space, allowing the model to infer the optimal number of topics directly from the data.

• Enables infinite topic mixtures that grow with corpus complexity
• Uses Chinese Restaurant Process as the underlying combinatorial process
• Particularly useful when the true topic cardinality is unknown

A Markov Chain Monte Carlo (MCMC) algorithm used for approximate posterior inference in Bayesian topic models. Gibbs sampling iteratively samples latent topic assignments for each word token conditioned on all other current assignments.

• Converges to the true posterior given sufficient iterations
• Computationally intensive for large corpora
• Contrasts with variational inference which optimizes a tractable lower bound
• Foundation for collapsed Gibbs sampling in LDA implementations like Gensim

An interactive visualization library for interpreting topic models by projecting inter-topic distances onto a two-dimensional plane. The intertopic distance map uses multidimensional scaling to display topic overlap and distinctness.

• Displays salient terms using a relevance metric balancing frequency and lift
• Allows users to adjust the relevance parameter λ to explore topic-word relationships
• Compatible with BERTopic through its embedding-based visualization approach
• Essential for qualitative model validation and stakeholder communication