Dual-Encoder Architecture

The architecture consists of two separate, identical neural networks—one for the query and one for the document. These towers operate in parallel and do not interact during encoding. This design allows for the massive pre-computation and indexing of all document embeddings offline, which is the foundation for fast retrieval. For edge RAG, this means the document index can be compiled, optimized, and stored on the device ahead of time.

Both the query and document encoders project their inputs into the same high-dimensional vector space. Semantic similarity is measured by the proximity of vectors in this space, typically using cosine similarity or dot product. The model is trained so that a query and a relevant document have a high similarity score. This shared space enables the use of fast Approximate Nearest Neighbor (ANN) search algorithms, which are critical for on-device performance.

Dual-encoders can be configured for different retrieval scenarios:

• Symmetric: Uses the exact same model weights for both query and document encoders. Ideal for tasks like paraphrase or duplicate detection.
• Asymmetric: Employs two different models (e.g., a lightweight model for queries, a heavier one for documents). This is common in search, where queries are short and documents are long, allowing for optimization of the query-side tower for edge inference speed.

These models are trained using a contrastive loss function, such as InfoNCE or multiple negatives ranking loss. The objective teaches the encoder to pull the embeddings of a positive (query, relevant document) pair closer together while pushing apart embeddings of negative (query, irrelevant document) pairs. This training creates a well-structured embedding space where semantic relevance translates to geometric closeness.

The separation of encoding and similarity calculation provides major efficiency gains:

• Query Encoding: A single, fast forward pass through the query encoder.
• Similarity Search: A highly optimized lookup against a pre-built index (e.g., HNSW, IVF). This decoupling avoids the quadratic complexity of cross-attention mechanisms, making real-time retrieval feasible on edge hardware with limited compute.

Dual-encoders are often the student model in a knowledge distillation pipeline. A larger, more accurate but slower cross-encoder (which performs deep interaction between query and document) acts as the teacher. The teacher's superior ranking knowledge is distilled into the dual-encoder student, allowing it to achieve higher accuracy than if trained on labels alone, while retaining its efficient two-tower architecture for edge deployment.

Dual-encoders are used to find candidate answers or to assess textual entailment by measuring the similarity between different text pairs in a shared semantic space.

• Open-Domain QA: Identifying potential answer-containing paragraphs from massive corpora like Wikipedia in response to a factoid question.
• Sentence Pair Classification: Determining if a hypothesis is entailed by, contradicts, or is neutral to a given premise (e.g., for MNLI benchmark).
• Duplicate Detection: Identifying near-duplicate questions on forums or semantically similar customer support tickets.

The dual-encoder framework extends beyond text-to-text retrieval to align different data modalities within a unified embedding space.

• Image-Text Retrieval: Encoding images and their captions separately (e.g., using a vision transformer and a text transformer) to enable searching images with text or generating captions for images via nearest neighbor lookup.
• Audio-Visual Search: Finding video clips based on a spoken query or sound effect.
• Product Search: Retrieving items from a catalog using a combination of image, text, and attribute data.

The separation of inference makes dual-encoders exceptionally suitable for edge AI and mobile applications where resources, latency, and privacy are paramount.

• Offline-Capable Search: Document embeddings are pre-computed and stored in a compact, optimized index (e.g., using HNSW or Product Quantization) on the device. Only the lightweight query encoder runs in real-time.
• Privacy-Preserving Search: Sensitive user queries never leave the device, as all retrieval happens locally against the on-device index.
• Hardware Optimization: The simple, parallel structure allows for efficient compilation and execution on mobile NPUs or via frameworks like TFLite and ONNX Runtime.

The primary training objective for dual-encoder models. It teaches the query and document encoders to produce similar embeddings for semantically related pairs (positives) and dissimilar embeddings for unrelated pairs (negatives).

• Key Mechanism: Uses a loss function like InfoNCE or triplet loss.
• Edge Relevance: Enables training of highly efficient, lightweight encoders that perform well with limited data, crucial for on-device models.
• Example: Training a model so the query "symptoms of flu" is close to a medical document about influenza, but far from a document about automobile repair.

The contrasting architecture to a dual-encoder. A cross-encoder passes the concatenated query and document through a single, deeper neural network (like BERT) to produce a relevance score.

• Key Difference: Achieves higher accuracy but is computationally expensive, as it must process every query-document pair at inference time.
• Practical Use: Often used as a teacher model in knowledge distillation to train a more efficient dual-encoder student model.
• Trade-off: Cross-encoders provide precision for reranking; dual-encoders provide speed for initial retrieval.

A dense, fixed-length vector representation of data (text, image, etc.) in a high-dimensional space. In a dual-encoder system, both queries and documents are mapped to embeddings.

• Shared Space: The core innovation is that both encoders project into the same vector space, where similarity is measured by distance (e.g., cosine similarity).
• Pre-computation: Document embeddings can be generated and indexed offline, which is the source of the dual-encoder's retrieval speed.
• Dimensionality: Typical embedding dimensions range from 384 to 768 for edge-optimized models, balancing representational power and storage cost.

A model compression technique where a large, high-accuracy teacher model (often a cross-encoder) transfers its ranking knowledge to a smaller, faster student model (a dual-encoder).

• Process: The student dual-encoder is trained to mimic the teacher's relevance scores or pairwise preferences.
• Outcome: The student achieves accuracy much closer to the teacher's while retaining the dual-encoder's efficient inference profile.
• Edge Impact: This is a primary method for creating high-quality, compact retrieval models suitable for deployment on edge hardware.

A synonymous term for dual-encoder. Both terms describe a architecture with two separate, parameter-independent encoders. 'Bi-encoder' emphasizes the two (bi) encoding processes.

• Usage Note: In academic literature, 'dual-encoder' and 'bi-encoder' are often used interchangeably.
• Nuance: Some contexts use 'bi-encoder' when the two encoders have identical architecture but are not weight-tied, and 'dual-encoder' as a more general umbrella term.
• Key Concept: Regardless of naming, the defining characteristic is the independent encoding of query and document for later similarity comparison.