Lexical Boosting

Lexical boosting increases the influence of sparse retrieval scores (like BM25) during hybrid fusion. When a query contains a rare token—such as a part number, error code, or legal citation—the system applies a higher weight to the lexical signal, ensuring documents containing the exact string are prioritized over semantically similar but textually distinct candidates. This prevents dense vector models from 'smoothing over' critical precision matches.

Unlike static fusion weights, lexical boosting operates dynamically at query time. The system analyzes incoming queries for lexical specificity triggers:

• Low document frequency terms: Words appearing in very few documents
• Numeric or alphanumeric codes: SKUs, serial numbers, error codes
• Exact phrase patterns: Quoted strings or boilerplate legal language When detected, the BM25 or sparse score component receives a multiplicative boost before fusion with dense scores.

Lexical boosting provides a tunable knob for managing the precision-recall spectrum. For queries where false positives are costly—such as regulatory document retrieval or medical code lookup—boosting lexical signals increases precision at the expense of recall. The technique is complementary to semantic boosting, which amplifies dense scores for conceptual queries. Together they enable query-intent-adaptive fusion.

Effective lexical boosting requires proper score normalization before fusion. Raw BM25 scores are unbounded and can dominate dense cosine similarity scores (typically in [0,1]). The boosting pipeline must:

• Apply min-max normalization to both score distributions
• Apply the lexical boost multiplier to the normalized sparse score
• Combine via weighted sum fusion or reciprocal rank fusion Without normalization, boosting can catastrophically suppress all semantic matches.

The primary use case for lexical boosting is exact identifier retrieval. Dense embedding models often fail on:

• Product SKUs: 'XJ-4492-BLK' has no semantic meaning
• Error codes: 'ERR_0x7B_FATAL' requires exact string match
• Legal citations: '42 U.S.C. § 1983' must match precisely Lexical boosting ensures these queries bypass semantic interpretation and retrieve documents containing the literal string, maintaining 100% recall on exact matches.

Lexical boosting functions as a soft fallback mechanism within hybrid search. When a query is classified as navigational or exact-match intent by a query intent classifier, the system can dynamically shift from a balanced fusion to a lexical-dominant configuration. This prevents the empty results problem where dense search returns semantically related but irrelevant documents for precise lookup queries, avoiding the need for a hard fallback to pure BM25.

The process of rescaling relevance scores from heterogeneous retrieval subsystems into a common, comparable range before fusion. Without normalization, a subsystem producing scores in the range [0, 1000] would dominate one producing scores in [0, 1]. Common techniques include:

• Min-Max Normalization: Rescales to [0, 1] using the observed min and max.
• Z-Score Normalization: Centers scores around zero based on mean and standard deviation.
• Sum-to-Unity: Divides each score by the sum of all scores.

A score aggregation method that multiplies the sum of normalized scores by the number of retrieval systems that returned the document. CombMNZ explicitly boosts documents found by multiple independent sources, operating on the principle that consensus across sparse and dense retrievers is a strong relevance signal. The formula is CombMNZ = (Σ normalized_scores) × count_of_systems_returning_doc.

An architectural pattern where sparse and dense retrieval execute independently against separate indexes, and result lists are merged only after both subsystems complete. Late fusion allows each retriever to be optimized in isolation but requires a robust fusion algorithm to reconcile potentially disjoint result sets. Contrast with early fusion, where embeddings are combined before a single search.

A precision-focused re-ranking stage applied after fusion. Unlike Bi-Encoders, which encode queries and documents independently, a Cross-Encoder processes the concatenated query-document pair through full Transformer self-attention. This joint processing captures fine-grained semantic interactions but is computationally expensive, so it is typically applied only to the top-K candidates from a faster first-stage retriever.

A preprocessing step that analyzes a query to determine its type—navigational, informational, or transactional—and dynamically adjusts fusion weights. For example, a navigational query containing a product code triggers lexical boosting to amplify BM25 scores, while an informational query like 'how to reduce cloud costs' shifts weight toward dense semantic retrieval.