Analyzer

Every analyzer operates through a strict sequential pipeline of three component types:

• Character Filters: Pre-process the raw text before tokenization. They can strip HTML markup, replace patterns with regex, or map characters like & to and.
• Tokenizer: The mandatory core that segments the character stream into discrete tokens. A standard tokenizer splits on whitespace and punctuation.
• Token Filters: Post-process the token stream. They apply lowercasing, stemming, stop word removal, and synonym expansion in a configurable order.

The output is an inverted index-ready stream of terms.

The tokenizer is the only mandatory component and defines how text is split into searchable units.

Common strategies include:

• Standard Tokenizer: Splits on whitespace and punctuation, removes most symbols. Good for general text.
• Whitespace Tokenizer: Splits only on whitespace, preserving punctuation within tokens.
• N-gram Tokenizer: Generates overlapping substrings of length n, enabling robust partial matching and typo tolerance.
• Pattern Tokenizer: Uses a regular expression to define token boundaries, ideal for structured data like log files.

The choice of tokenizer fundamentally determines what constitutes a matchable term in your index.

Token filters transform the token stream to improve recall and precision. They execute in a defined order:

• Lowercase Filter: Normalizes case so 'Apple' and 'apple' match identically.
• Stemmer: Reduces words to their root form. The Porter Stemmer maps 'running' to 'run' and 'fishing' to 'fish', collapsing morphological variants.
• Stop Words Filter: Removes high-frequency, low-discriminative words like 'the', 'is', and 'at' to reduce index size.
• Synonym Filter: Expands tokens with equivalents, so a query for 'car' also matches documents containing 'automobile'.
• Shingle Filter: Combines adjacent tokens into multi-word phrases, enabling phrase matching without positional queries.

Analyzers are applied at two distinct moments with different goals:

• Index-Time Analysis: Processes documents as they are ingested. The resulting tokens are stored in the inverted index. This analysis should be thorough—applying stemming, lowercasing, and stop word removal.
• Search-Time Analysis: Processes the user's query string. This must be aligned with index-time analysis but can be lighter. For example, you might skip stemming on the query if you want exact form matching.

Mismatched analyzers between index and search time are a primary cause of vocabulary mismatch and zero-result queries.

Search engines ship with pre-configured analyzers for common languages and use cases:

• Standard Analyzer: General-purpose, uses standard tokenizer + lowercase filter.
• Simple Analyzer: Splits on non-letter characters and lowercases.
• Whitespace Analyzer: Splits on whitespace only, no further filtering.
• Language Analyzers: Specialized for English, French, Chinese, etc., with language-specific stemming and stop words.

Custom Analyzers are defined when you need precise control—combining a pattern tokenizer with a synonym filter and a custom stemmer override, for example. This is essential for domain-specific vocabularies like medical or legal text.

Most search platforms provide an Analyze API endpoint that lets you inspect exactly how text is processed before indexing.

You submit a text string and the analyzer name, and the API returns:

• The final list of tokens that will be indexed or searched.
• The position of each token in the stream.
• The start and end offsets in the original text.

This is the definitive debugging tool for diagnosing relevance issues. If a query isn't matching expected documents, the Analyze API reveals whether the tokenizer split a compound word incorrectly or a stemmer reduced a term too aggressively.

The first stage in the analysis chain that segments a raw character stream into discrete tokens. A tokenizer defines the rules for word boundaries, such as splitting on whitespace and punctuation.

• Standard Tokenizer: Splits on word boundaries and removes most punctuation.
• Whitespace Tokenizer: Splits only on whitespace characters.
• Pattern Tokenizer: Uses a regular expression to capture tokens.

The choice of tokenizer directly impacts how the subsequent filters process the text. For example, a standard tokenizer will break [email protected] into user and domain.com, while a custom pattern might preserve the email as a single token.

An algorithm that reduces inflected or derived words to their root form to improve recall. Stemming is a heuristic process that chops off prefixes or suffixes, often producing a non-dictionary stem.

• Porter Stemmer: A widely used algorithm for English that applies a sequence of reduction rules.
• Snowball Stemmer: An improved, multi-lingual framework for creating stemming algorithms.
• KStem: A less aggressive stemmer that minimizes over-stemming errors.

Stemming maps running, runs, and ran to run, ensuring a query for one variant matches all others. This is distinct from lemmatization, which uses a vocabulary and morphological analysis to return the dictionary base form.

A filter that removes high-frequency, low-discrimination words from the token stream. Words like the, is, at, and which appear in nearly every document and provide negligible signal for relevance ranking.

• Default Lists: Most search engines ship with a standard stop word list for common languages.
• Custom Stop Words: Domain-specific noise words can be added, such as company in a corporate intranet.
• Trade-off: Removing stop words reduces index size and query latency but can break phrase queries like to be or not to be or searches for bands named The Who.

Modern analyzers often prefer to keep stop words and rely on scoring functions like BM25 to naturally downweight them via inverse document frequency.

A normalization filter that converts all tokens to lowercase to enable case-insensitive matching. This is one of the simplest yet most critical filters in an analyzer pipeline.

• Purpose: Ensures that a query for Apple matches documents containing apple.
• Placement: Always applied after tokenization and before stemming.
• Exceptions: Proper nouns like acronyms or camelCase identifiers may require a custom analyzer that preserves case for specific patterns.

Without this filter, a search for Machine Learning would fail to match a document containing machine learning, severely degrading recall.

A filter that expands tokens to include their synonyms, bridging the vocabulary mismatch gap between query terms and document terms. This is a form of query expansion applied at index time or query time.

• Index-Time Synonyms: Synonyms are written into the index, increasing recall but also index size.
• Query-Time Synonyms: The query is expanded, keeping the index lean but increasing query complexity.
• Multi-Word Synonyms: Rules can map phrases like machine learning to ML or artificial intelligence.

A well-maintained synonym file is essential for e-commerce search, where a query for sneakers must match products listed as running shoes.

A filter that generates overlapping substrings of length n from each token. This technique enables robust partial matching, autocomplete suggestions, and typo-tolerant search.

• Edge N-gram: Generates n-grams only from the start of a token, ideal for search-as-you-type functionality.
• N-gram: Generates all possible substrings, useful for substring matching and handling compound languages.
• Shingle: A word-level n-gram that captures multi-word phrases, improving proximity-aware relevance.

For example, an edge n-gram filter with min=2 and max=5 applied to analyzer produces an, ana, anal, analy, allowing a query for ana to match immediately.