Synonym Filter

A synonym filter operates on a stream of tokens produced by a tokenizer, expanding specific tokens into multiple alternatives at the same position. This ensures that a query for 'car' also matches documents containing 'automobile' without altering the original token's position. The expansion happens during both indexing and query time, though modern best practices often apply it asymmetrically to avoid index bloat.

Advanced filters handle multi-word synonyms, mapping a single token to a phrase or vice versa. For example, the token 'nyc' can be expanded to 'new york city' at query time. This requires sophisticated graph-based token stream management to ensure the phrase is treated as a single semantic unit, preventing accidental partial matches against unrelated documents.

Applying synonyms at index time increases recall but inflates the inverted index and removes the ability to change synonyms without re-indexing. Applying them at query time keeps the index lean and allows dynamic updates but increases query latency. A hybrid approach often applies aggressive expansion at query time while keeping the index normalized to balance precision and performance.

A naive synonym filter can destroy precision by expanding ambiguous terms. For example, expanding 'apple' to 'fruit' and 'macintosh' simultaneously introduces noise. Sophisticated implementations integrate with Named Entity Recognition or use context-aware rules to apply synonyms only when the surrounding tokens indicate the correct domain, preventing catastrophic query drift.

The top-level component in a search engine that processes text into a stream of tokens. It orchestrates a tokenizer followed by a chain of token filters—including the synonym filter—to create searchable terms. The analyzer defines the entire text processing pipeline for a specific field.

A module that segments a raw character stream into discrete tokens, typically words, based on configurable rules like whitespace and punctuation boundaries. The tokenizer runs before any synonym filter, meaning synonym expansion operates on the token stream, not the raw text.

An algorithm that reduces inflected or derived words to their root form, such as mapping 'running' to 'run'. When combined with a synonym filter, stemming is typically applied after synonym expansion to ensure that both the original token and its synonyms are reduced to a consistent morphological base for improved recall.

High-frequency words like 'the' and 'is' that are filtered out during text analysis because they carry little discriminative value. A synonym filter typically processes the token stream after stop word removal to avoid wasting computational resources on expanding terms that will be discarded and to prevent spurious synonym matches on noise words.

The fundamental retrieval problem that synonym filters are designed to solve. It occurs when a relevant document uses different words to describe a concept than the words used in the query, causing a lexical matching failure. A synonym filter bridges this gap by mapping 'car' to 'automobile', ensuring the query matches documents regardless of the specific term chosen by the author.