Long Document Re-Ranking

A technique that segments a long document into overlapping passages of fixed length (e.g., 512 tokens with a 256-token stride). Each passage is scored independently against the query using a Cross-Encoder, and the final document score is derived by aggregating these passage-level scores, typically via max-pooling or mean-pooling. This ensures no contextual boundary artificially splits a relevant answer across two chunks.

A passage score aggregation function where the final document relevance score is the maximum score among all its constituent sliding window passages. This strategy operates on the assumption that a document is as relevant as its most relevant passage. It is highly effective for factoid queries where the answer is localized to a single paragraph, preventing dilution from irrelevant sections.

A two-stage approach designed for very long documents (e.g., legal contracts or research papers). The document is first divided into large structural segments (chapters, sections). A lightweight Bi-Encoder or sparse retriever selects the top-k most relevant segments, and only these segments are passed through the computationally expensive Cross-Encoder for fine-grained scoring. This drastically reduces the number of inference calls compared to exhaustive sliding windows.

A simple baseline strategy that fits a long document into the Cross-Encoder's maximum sequence length by retaining the first N tokens and the last M tokens, discarding the middle. This leverages the common structure of documents where introductions and conclusions contain dense summaries of the content. While computationally cheap, it risks missing critical evidence buried in the body of the text.

A pre-filtering step that uses a sparse lexical model like BM25 to score individual sentences or passages within a long document against the query before invoking the Cross-Encoder. Only the top-k BM25 passages are concatenated and fed to the Cross-Encoder. This combines the exact-match precision of sparse retrieval with the semantic understanding of a neural re-ranker, optimizing for both latency and relevance.

An alternative to max-pooling where the final document score is the arithmetic mean of all passage-level Cross-Encoder scores. This strategy is more suitable for broad, topical queries where overall document relevance matters more than a single localized answer. It penalizes documents that are only partially relevant, favoring those with consistently high semantic alignment across all sections.

The primary strategy for scoring documents longer than the Cross-Encoder's maximum sequence length (typically 512 tokens). The document is segmented into overlapping or contiguous windows of fixed token length. Each window is independently scored against the query using a Cross-Encoder, producing a set of passage-level relevance scores. The final document score is derived by aggregating these window scores, most commonly via max-pooling to capture the highest relevance signal anywhere in the document. Window overlap ensures critical context spanning segment boundaries is not lost.

A score aggregation function that selects the maximum relevance score from all sliding window passages as the single document-level score. The intuition is that a document's relevance to a query is determined by its most relevant passage. Mathematically: score(doc) = max(score(window_1), score(window_2), ..., score(window_n)). This method is robust to documents where the answer is concentrated in one section but risks over-weighting documents with a single highly-scored but out-of-context snippet.

A multi-stage approach that first scores individual passages, then feeds the top-k passage representations into a secondary aggregation model. This can be implemented as:

• Mean-pooling of top-k passage embeddings before a final linear layer
• A lightweight Transformer aggregator that attends over passage [CLS] tokens
• Weighted sum where weights are learned based on passage position This method captures the document's global relevance distribution rather than relying on a single passage, improving precision for queries requiring synthesis across sections.

A systematic failure mode where a Cross-Encoder assigns disproportionately high scores to the document's opening passage regardless of query relevance. This occurs because pre-training corpora often position key information at the beginning. Mitigation strategies include:

• Random window shuffling during training to decorrelate position from relevance
• Position-aware scoring heads that learn to down-weight opening passage bias
• Calibration on passage position using a held-out validation set to normalize scores by segment location

The engineering discipline of maximizing relevance signal within the Cross-Encoder's fixed 512 or 4096 token context window. Techniques include:

• Query compression via extractive summarization to reserve tokens for document content
• Document truncation strategies that prioritize sections with high BM25 or Bi-Encoder alignment
• Dynamic window sizing where passage length adapts based on document structure (e.g., respecting paragraph boundaries)
• Stride length tuning to balance overlap redundancy against computational cost

An alternative to sliding window Cross-Encoders that uses ColBERT-style MaxSim scoring across the full document. The document is encoded as a large set of token-level embeddings (potentially thousands of tokens), and the query tokens interact with all document tokens via maximum similarity. This avoids the window boundary problem entirely and enables fine-grained token matching across the entire document length. The trade-off is increased storage for per-token document embeddings and higher retrieval latency compared to single-vector Bi-Encoders.