Feature Extraction

Represents text as an unordered multiset of words, discarding grammar and word order entirely. Each document becomes a vector where dimensions correspond to vocabulary terms and values represent occurrence counts.

• Mechanism: Tokenizes text, builds a global vocabulary, then counts term frequencies per document
• Output: A sparse, high-dimensional vector of length |V| (vocabulary size)
• Key limitation: Treats 'dog bites man' and 'man bites dog' as identical vectors
• Use case: Baseline for text classification when semantic nuance is not critical

Refines BoW by down-weighting terms that appear frequently across all documents while up-weighting terms that are discriminative for a specific document.

• Term Frequency (TF): Raw count or log-normalized frequency of term t in document d
• Inverse Document Frequency (IDF): log(N / df_t), where df_t is the number of documents containing t
• TF-IDF = TF × IDF: High weight for terms that are frequent in few documents
• Advantage over BoW: Reduces the influence of stop words and corpus-wide common terms without explicit filtering

Maps each token to a binary vector with a single '1' at the index corresponding to that token's position in the vocabulary, and '0' everywhere else.

• Vector length: Equal to vocabulary size |V|
• Result: An orthogonal representation where every token is equidistant from every other token
• Critical flaw: Contains zero semantic information—'king' and 'queen' are as dissimilar as 'king' and 'toaster'
• Modern role: Used as input labels for classification heads, not as document representations

Extends BoW by treating contiguous sequences of N tokens as individual features, capturing local word order and short phrases.

• Unigram (N=1): Single words—standard BoW
• Bigram (N=2): Adjacent pairs like 'New York' or 'machine learning'
• Trigram (N=3): Three-word sequences like 'out of vocabulary'
• Trade-off: Captures more context but causes combinatorial explosion in feature space; vocabulary grows exponentially with N
• Mitigation: Apply frequency thresholds to prune rare n-grams

Applies a hash function to map tokens directly to column indices in a fixed-size vector, eliminating the need to store a vocabulary in memory.

• Stateless: No vocabulary dictionary required; can process streaming data
• Collisions: Multiple tokens may hash to the same index, introducing noise—mitigated by using a large feature dimension (e.g., 2^18 to 2^22)
• Signed hashing: Uses the sign of the hash to cancel out collision noise, improving accuracy
• Best for: Large-scale, distributed, or online learning scenarios where vocabulary storage is prohibitive

Builds a BoW or n-gram representation but applies frequency-based filtering to reduce dimensionality and noise.

• max_features: Keep only the top K most frequent terms
• min_df: Ignore terms appearing in fewer than N documents (float for proportion)
• max_df: Ignore corpus-wide frequent terms above a threshold (corpus-specific stop words)
• Result: A compact, discriminative vocabulary that balances signal and dimensionality
• Implementation: Standard in scikit-learn's CountVectorizer and TfidfVectorizer

A foundational text vectorization model that represents a document solely as an unordered multiset of its words, completely discarding grammar and word order.

• Mechanism: Constructs a vocabulary of known words and scores each document by the presence or frequency of these words.
• Sparsity: Produces a high-dimensional, sparse vector where most values are zero.
• Limitation: Treats 'dog bites man' and 'man bites dog' as identical vectors, losing all sequential context.

A statistical measure used to evaluate how important a word is to a document in a collection. It balances local frequency against global rarity.

• Term Frequency (TF): Measures how often a term appears in a specific document.
• Inverse Document Frequency (IDF): Diminishes the weight of terms that appear frequently across the entire corpus, such as stop words.
• Formula: TF-IDF = TF * log(N/df), where N is the total number of documents and df is the document frequency of the term.

The fundamental preprocessing step of segmenting a continuous string of raw text into discrete units called tokens, which serve as the atomic input for feature extraction.

• Word Tokenization: Splitting text by whitespace and punctuation.
• Subword Tokenization: Algorithms like Byte-Pair Encoding (BPE) break words into smaller, frequent sub-units to handle out-of-vocabulary terms.
• Impact: The tokenization strategy directly defines the initial vocabulary size and granularity of the feature space.

The process of removing high-frequency, low-semantic-value words from a text corpus before vectorization to reduce noise and dimensionality.

• Examples: 'the', 'is', 'at', 'which'.
• Mechanism: Uses a predefined list to filter tokens, preventing these words from dominating frequency-based models like TF-IDF.
• Trade-off: Modern contextual models often retain stop words as they carry syntactic structure, but they remain critical for sparse feature extraction.

Two distinct normalization techniques used to reduce inflectional forms to a common base form, collapsing the feature space for improved recall.

• Stemming: A crude, rule-based heuristic that chops off affixes (e.g., 'running' → 'run', 'studies' → 'studi'). The Porter Stemmer is a classic example.
• Lemmatization: A precise, vocabulary-based process that returns the dictionary form using morphological analysis (e.g., 'better' → 'good').
• Use Case: Stemming is faster for search; lemmatization is preferred when semantic accuracy matters.

A contiguous sequence of N items from a text sample used to capture local word order and context that a pure Bag-of-Words model loses.

• Unigram (N=1): Single words.
• Bigram (N=2): Pairs of consecutive words (e.g., 'New York').
• Trigram (N=3): Triplets of words.
• Trade-off: Increases feature dimensionality exponentially but captures critical phrases like 'not good' that unigrams would misinterpret.