Lexical Matching

The core principle is surface-form matching: a document is retrieved only if it contains the exact character sequence specified in the query. This is implemented via an inverted index, which maps each unique term to a postings list of document identifiers. The process is binary and deterministic—there is no learned similarity function, only set intersection between query terms and indexed vocabulary.

The fundamental failure mode of lexical matching. A document may be highly relevant but uses different terminology than the query. For example, a query for 'heart attack' will miss documents discussing 'myocardial infarction' unless synonym expansion is applied. This gap between user intent and document vocabulary is the primary motivation for semantic search and dense retrieval methods.

Matching fidelity depends entirely on how text is segmented into tokens. The analyzer pipeline—comprising a tokenizer, lowercase filter, stemmer, and stop-word removal—determines which surface forms become indexable terms. A query for 'running' will match 'run' only if a stemmer reduces both to the same root. Inconsistent tokenization across indexing and query time causes silent retrieval failures.

Lexical matching supports structured constraints beyond simple term overlap. Boolean retrieval uses AND, OR, and NOT operators to enforce logical conditions on term presence. Phrase matching requires terms to appear in exact adjacency and order, using positional data stored in the postings list. These constraints add precision but reduce recall by excluding documents that discuss the same concepts with different syntax.

Pure overlap is insufficient for ranking. Algorithms like TF-IDF and BM25 weight each match by its statistical significance. Term frequency boosts documents where a query term appears often, while inverse document frequency downweights common words like 'the'. BM25 adds saturation to prevent term frequency from dominating and document length normalization to avoid bias toward longer documents.

In lexical matching, documents and queries are encoded as sparse vectors in a high-dimensional space where each dimension corresponds to a vocabulary term. Most dimensions are zero. This sparsity enables efficient retrieval via inverted indices, avoiding the computational cost of dense vector comparisons. The trade-off is that these vectors capture only term presence, not semantic meaning or word order.