Bag-of-Words (BoW)

BoW explicitly discards word order and grammar, treating a document as an unstructured collection of its words. The sentence 'The cat sat on the mat' is represented identically to 'The mat sat on the cat' in a pure BoW model. This simplification trades syntactic information for computational efficiency, making it highly effective for tasks like document classification and topic modeling where the presence of specific terms is more predictive than their sequence.

A BoW model constructs a fixed-length vector for each document based on a predefined vocabulary derived from the entire corpus. Each dimension corresponds to a unique token. The value in that dimension is typically the term frequency (TF)—a raw count of occurrences—or a weighted variant like TF-IDF. This transforms unstructured text into a numerical matrix suitable for machine learning algorithms.

• Sparse representation: Most entries are zero.
• Dimensionality: Equal to the size of the corpus vocabulary.

The resulting document-term matrix is inherently high-dimensional and sparse. Because any single document uses only a tiny fraction of the total corpus vocabulary, the matrix is dominated by zeros. This sparsity is a key computational consideration.

• Memory efficiency: Requires sparse matrix formats like CSR (Compressed Sparse Row).
• Curse of dimensionality: High dimensionality can impact distance metrics and model performance.
• Contrast: Dense embedding models like Word2Vec or BERT produce low-dimensional, dense vectors.

The primary limitation of BoW is the complete loss of word order and semantic context. It cannot distinguish between 'not good' and 'good', nor capture polysemy—the word 'bank' in 'river bank' and 'savings bank' is treated identically. This limitation is addressed by modern dense retrieval and transformer-based architectures that model context. BoW remains relevant as a strong baseline and for applications where vocabulary presence is the dominant signal.

To partially mitigate the loss of word order, BoW models are often extended to include n-grams—contiguous sequences of N tokens. A bigram (N=2) model captures adjacent pairs like 'New York' or 'machine learning', preserving some local context. This increases vocabulary size exponentially but captures critical collocations and short phrases that carry meaning distinct from their individual words.

The quality of a BoW representation is highly dependent on the preprocessing pipeline. Critical steps include:

• Tokenization: Segmenting text into discrete units.
• Case folding: Converting to lowercase to merge 'Apple' and 'apple'.
• Stop word filtering: Removing high-frequency, low-information words like 'the' and 'is'.
• Stemming/Lemmatization: Reducing words to a base form to consolidate 'running', 'ran', and 'runs'.

A direct evolution of the BoW model that weights terms by their discriminative power. Unlike raw counts, TF-IDF penalizes words that appear frequently across all documents (like 'the' or 'and') while rewarding terms that are frequent in a specific document but rare in the general corpus. This addresses BoW's key weakness: treating all words as equally important. The resulting sparse vectors are far more effective for search relevance than simple frequency counts.

A technique that extends the BoW model to capture local word order. While standard BoW discards sequence entirely, N-grams treat contiguous sequences of N items as single tokens. For example, a bigram model would represent 'New York' as a distinct feature rather than two separate unigrams. This partially recovers context lost by the independence assumption, though at the cost of exploding feature dimensionality.

A preprocessing step almost always paired with BoW to reduce dimensionality and noise. High-frequency function words like 'the', 'is', and 'at' carry minimal semantic content and dominate raw term frequency counts. Filtering them out before constructing the BoW vector dramatically reduces the feature space size and often improves downstream model accuracy by focusing the representation on content-bearing terms.

Two normalization strategies that collapse morphological variants into a single feature, directly addressing BoW's vocabulary sparsity problem. Stemming uses crude heuristic rules to chop affixes (e.g., 'running' → 'run'), while lemmatization uses a dictionary to return the canonical base form (e.g., 'better' → 'good'). Without this step, 'run', 'runs', and 'running' would be treated as three unrelated features, fragmenting the signal.

The modern neural alternative to sparse BoW representations. Instead of high-dimensional, mostly-zero vectors, DPR uses a dual-encoder transformer architecture to embed text into dense, low-dimensional vectors (e.g., 768 dimensions). These embeddings capture semantic similarity rather than lexical overlap, meaning 'car' and 'automobile' are close in vector space despite sharing no characters—a fundamental limitation BoW cannot overcome.

The critical first step that defines the vocabulary of a BoW model. Simple whitespace tokenization fails on punctuation and contractions. Modern approaches like Byte-Pair Encoding (BPE) break text into subword units, allowing the model to represent out-of-vocabulary words as compositions of known fragments. The choice of tokenizer directly determines the granularity and size of the resulting BoW feature space.