Stop Words

Stop words are almost exclusively function words—the grammatical glue of a language—rather than content words that carry the core meaning.

• Function words (stopped): determiners (the, a), prepositions (in, of), conjunctions (and, but), auxiliary verbs (is, was)
• Content words (kept): nouns, main verbs, adjectives, adverbs

This distinction is why removing 'the' rarely changes the topic of a document, but removing 'blockchain' destroys it.

Stop words dominate text frequency following Zipf's Law: the most frequent word occurs roughly twice as often as the second, three times as often as the third, and so on.

• In English, 'the' alone accounts for ~7% of all tokens in a typical corpus
• The top 10 words (the, of, and, a, to, in, is, you, that, it) can represent 25-30% of all tokens
• Removing these drastically reduces index size without affecting recall for most queries

The primary motivation for stop word removal in sparse retrieval systems like BM25 is computational, not linguistic.

• Inverted index size reduction: Eliminating stop words can shrink the postings list by 30-40%
• Query latency improvement: Fewer terms to look up and score means faster retrieval
• Memory footprint: Smaller indices fit in RAM, reducing infrastructure costs

This optimization was critical in the era of limited storage, but is less relevant for dense vector search where embeddings encode all tokens.

Stop word removal is not harmless. It creates a measurable precision-recall trade-off that must be evaluated per domain.

• Phrasal queries break: 'to be or not to be' becomes empty; 'The Who' loses its referent
• Negation distortion: Removing 'not' from 'not guilty' inverts meaning entirely
• Rare entity collisions: The film 'It' or the band 'The The' become unsearchable

Modern systems often retain stop words in the index but downweight them using TF-IDF or BM25 saturation rather than removing them outright.

A universal stop word list is a myth. Effective filtering requires domain adaptation.

• General English: NLTK's list of 179 words, or the Snowball list of 174
• Legal text: 'heretofore', 'whereas', 'aforementioned' carry structural meaning
• Medical records: 'patient', 'diagnosis' are too frequent to be discriminative
• E-commerce: 'product', 'price', 'shipping' may need retention for query intent

Building a custom list involves calculating collection frequency and manually reviewing high-frequency terms for semantic value.

In dense retrieval and transformer-based models, stop word removal is largely obsolete and often harmful.

• Contextualization: BERT and similar models use stop words to build syntactic structure and disambiguate meaning
• Attention mechanisms: Function words help establish relationships between content words across long distances
• Query-document interaction: Cross-encoders need the full grammatical signal to score relevance accurately

Modern tokenizers like BPE and WordPiece retain all tokens, letting the model learn which to attend to rather than pre-filtering.