Cross-Encoder

A bi-encoder is a neural architecture where the query and document are encoded independently into separate dense vector embeddings. This enables efficient similarity search via pre-computed indexes but can be less precise than joint modeling.

• Key Advantage: Enables pre-computation of all document embeddings, allowing for millisecond-level retrieval via vector similarity search (e.g., cosine similarity).
• Trade-off: The independent encoding can struggle with complex semantic interactions that require deep, token-level comparison between query and document.

Reranking is a two-stage retrieval pipeline where a large set of candidate documents is first retrieved (e.g., via a fast bi-encoder or keyword search) and then re-scored by a more powerful, computationally expensive model to improve final ranking precision.

• Cross-Encoder Role: Acts as the perfect reranker due to its high accuracy in judging pairwise relevance.
• Pipeline Example: Retrieve 100 candidates with a vector database → rerank the top 20 with a cross-encoder → return the top 3 results.

Dense retrieval is a neural search paradigm where queries and documents are encoded into dense, low-dimensional vector embeddings (typically 384-768 dimensions). Relevance is determined by the similarity (e.g., cosine) between these embeddings.

• Contrast with Sparse Retrieval: Uses continuous semantic vectors instead of sparse keyword-based vectors (e.g., BM25).
• Relation to Cross-Encoder: Cross-encoders are not used for the initial dense retrieval phase due to speed constraints but are often used to train the bi-encoders that power it via knowledge distillation.

ColBERT (Contextualized Late Interaction over BERT) is a retrieval model that uses a late interaction mechanism. It encodes queries and documents into fine-grained token-level embeddings and computes similarity via a sum of maximum similarity scores across tokens.

• Architecture Bridge: Occupies a middle ground between bi-encoders (fast, independent encoding) and cross-encoders (slow, joint encoding).
• Efficiency: Allows for some pre-computation of document token embeddings, making it significantly faster than a cross-encoder while often being more expressive than a standard bi-encoder.

Retrieval-Augmented Generation (RAG) is an architecture that enhances an LLM's responses by retrieving relevant information from an external knowledge source and conditioning the generation on that context.

• Cross-Encoder Application: A cross-encoder is a critical component in advanced RAG systems for the reranking stage. After a vector database retrieves initial passages, a cross-encoder selects the most relevant ones to include in the LLM's limited context window, dramatically improving answer quality and reducing hallucinations.

Maximum Inner Product Search (MIPS) is the core computational problem of finding the data points whose vector representations yield the highest dot product (inner product) with a query vector.

• Foundation for Bi-Encoder Retrieval: When bi-encoder embeddings are normalized, cosine similarity is equivalent to the dot product. Thus, vector databases performing MIPS are the engine for fast bi-encoder retrieval.
• Contrast with Cross-Encoder: A cross-encoder solves a different problem: it computes a relevance score through deep, joint processing, not a simple dot product between independent vectors.