Reranking Latency

The choice of reranking model fundamentally dictates latency. Cross-encoder architectures, which perform full joint encoding of the query and document, have quadratic complexity (O(n²)) due to the self-attention mechanism over the concatenated sequence. This is the primary latency driver compared to faster bi-encoders or late-interaction models like ColBERT. Larger model sizes (e.g., 110M vs. 440M parameters) linearly increase compute time. Architectures using sparse attention (e.g., Longformer) can mitigate this for longer texts.

The reranking depth (k)—the number of initial candidates passed to the model—directly scales latency. Processing 100 candidates is ~100x slower than processing 1. This creates a key trade-off:

• High k (e.g., 100-200): Maximizes recall but incurs high latency.
• Low k (e.g., 10-20): Minimizes latency but risks missing relevant documents deeper in the initial ranking. Optimal k is determined by the recall-latency requirements of the specific application.

Transformer inference time scales with sequence length. For reranking, the total processed length is len(query) + len(document). Long documents or queries increase latency proportionally. Tokenization overhead is non-trivial; models with subword tokenizers (e.g., WordPiece for BERT, SentencePiece for T5) must process each candidate. Strategies to manage this include:

• Document chunking into fixed-size passages.
• Dynamic truncation based on model's maximum context window.
• Caching tokenized representations for static document corpora.

Latency is a function of the underlying hardware and software optimizations. Key factors include:

• GPU/Accelerator Memory: Limits batch size; smaller batches increase overhead.
• Inference Frameworks: Optimized runtimes like ONNX Runtime, TensorRT, or vLLM can provide 2-5x speedups over naive PyTorch inference.
• Quantization: Using INT8 or FP16 precision reduces memory bandwidth and increases compute throughput.
• Continuous Batching: Dynamically batches incoming requests of varying lengths to improve GPU utilization.
• Model Pruning & Distillation: Smaller student models distilled from large cross-encoders can approximate performance with significantly lower latency.

Latency isn't just model inference. System-level operations add overhead:

• Network I/O: Transferring candidate texts and scores between retrieval service, reranking service, and application.
• Pre/Post-processing: Tokenization, score normalization, and sorting final rankings.
• Microservice Communication: In containerized deployments, inter-service calls (gRPC, HTTP) introduce milliseconds of latency.
• Cold Starts: Serverless deployments or autoscaling can cause significant latency spikes when instantiating new model containers.
• Caching Strategies: Implementing caches for frequent query-document pairs or model outputs can dramatically reduce average latency for repetitive requests.

Processing candidates in batches is essential for throughput but complex for reranking due to variable-length sequences. Inefficient batching is a major latency source.

• Native Padding: The standard approach pads all sequences in a batch to the length of the longest sequence, wasting FLOPs on padding tokens.
• Optimized Batching: Techniques like bucket batching group similar-length sequences, or using frameworks with padding-free attention (e.g., Hugging Face's padding='max_length' vs. padding=True).
• Batch Size Trade-off: Larger batches improve GPU utilization but increase memory pressure and per-request latency if waiting to fill a batch. Dynamic batching strategies are critical for production systems.

The overarching pipeline architecture where reranking latency is incurred. A fast first-stage retriever (e.g., BM25, a bi-encoder) fetches a broad set of candidates (e.g., 100-1000 documents). This candidate set is then passed to a slower, more accurate second-stage model, typically a cross-encoder reranker, for final precision ordering. The total system latency is the sum of the first-stage retrieval time and the reranking latency for the candidate set.

The fundamental computational bottleneck causing high reranking latency. In a standard transformer-based cross-encoder, the self-attention mechanism scales with O(n²), where n is the combined sequence length of the query and document. Processing a single long document (e.g., 512 tokens) with a query can be expensive; processing k candidates multiplies this cost. This complexity is the primary reason rerankers are applied to a limited reranking depth (k) rather than the entire corpus.

The number of top candidates passed from the first-stage retriever to the reranker. This is the most direct lever for controlling reranking latency.

• Lower k (e.g., 10-50): Reduces latency and cost but risks missing a relevant document that the first-stage retriever ranked poorly.
• Higher k (e.g., 100-200): Increases recall potential but incurs linear increases in latency and compute. Tuning k involves a precision-recall-latency trade-off specific to the application's tolerance for delay.

A suite of techniques to reduce the computational cost and thus the latency of the reranking model itself. Key methods include:

• Model Quantization: Reducing the numerical precision of model weights (e.g., from FP32 to INT8) to speed up computation and reduce memory bandwidth.
• Pruning: Removing redundant neurons or weights from the model.
• Kernel Optimization: Using specialized libraries like ONNX Runtime or TensorRT for efficient GPU execution.
• Batching: Processing multiple query-document pairs simultaneously to amortize overhead, though this can increase per-request latency.

The core architectural trade-off defining the latency-accuracy frontier.

• Bi-Encoder: Encodes the query and document independently into dense vectors (embeddings). Relevance is a simple cosine similarity. Extremely low latency, allowing search over millions of documents in milliseconds, but typically lower accuracy than cross-encoders.
• Cross-Encoder: Jointly encodes the query and document, enabling deep token-level interaction. Much higher accuracy but with high latency due to quadratic complexity. Reranking pipelines use a bi-encoder for first-stage retrieval and a cross-encoder for second-stage reranking.

A technique to mitigate reranking latency by creating a faster, student model that approximates a slower, teacher model. A large, accurate cross-encoder teacher is used to generate relevance scores or preferences on a dataset. A smaller, more efficient student model (often a bi-encoder or a tiny transformer) is then trained to mimic these scores. The distilled student can be deployed for lower-latency reranking or even first-stage retrieval, preserving much of the teacher's accuracy at a fraction of the cost.