Evidence Retrieval

Uses dual-encoder architectures to map both queries and documents into a shared dense vector space. This enables semantic matching beyond keyword overlap.

• Bi-encoder model: Encodes queries and passages independently for fast maximum inner product search
• Approximate Nearest Neighbor (ANN): Uses FAISS or ScaNN indexes to retrieve top-k results in milliseconds
• Training objective: Contrastive loss with in-batch negatives, where relevant passages are pulled closer and non-relevant pushed apart

DPR achieves high recall by understanding paraphrased queries, making it essential when claims use different terminology than evidence documents.

Traditional bag-of-words retrieval using inverted indexes remains critical for exact term matching. BM25, the probabilistic ranking function, excels when claims contain rare entities or precise technical terms.

• Term frequency saturation: Prevents common words from dominating relevance scores
• Document length normalization: Penalizes overly long documents to avoid bias
• Inverted index efficiency: Enables sub-millisecond lookup across billions of documents

Hybrid systems combine BM25 scores with dense retrieval scores through linear interpolation or reciprocal rank fusion to capture both lexical precision and semantic breadth.

After initial retrieval, a cross-encoder model processes the query and candidate passage jointly through full self-attention. This provides fine-grained relevance scoring that bi-encoders miss.

• Joint encoding: Concatenates query and passage with a [SEP] token for deep interaction
• Relevance score: Outputs a single logit representing entailment probability
• Computational cost: Too expensive for full corpus search, applied only to top-100 or top-1000 candidates

Models like monoT5 and cross-encoder/ms-marco-MiniLM-L-6-v2 are standard rerankers that significantly boost precision at top ranks.

How documents are chunked directly impacts retrieval quality. Passage segmentation must balance context completeness against retrieval granularity.

• Fixed-length chunking: Splits documents at token limits (e.g., 256 tokens) with configurable overlap to prevent mid-sentence breaks
• Sentence-boundary segmentation: Uses NLP sentence tokenizers to create semantically coherent units
• Hierarchical indexing: Stores both sentence-level and paragraph-level chunks, retrieving at multiple granularities
• Sliding window with stride: Creates overlapping passages to ensure no evidence spans are split across boundaries

Poor segmentation is a leading cause of retrieval failure, as critical context may be truncated or diluted.

Raw claims are rarely optimal search queries. Query rewriting transforms a claim into a retrieval-friendly form before vector or lexical search.

• Doc2Query: Generates synthetic queries from documents during indexing to bridge vocabulary gaps
• Claim decomposition: Breaks multi-faceted claims into atomic sub-queries for independent retrieval
• Pseudo-relevance feedback: Uses top initial results to extract expansion terms for a second retrieval pass
• HyDE (Hypothetical Document Embeddings): Generates a hypothetical answer document from the claim, then retrieves real documents similar to that hypothetical

These techniques address the vocabulary mismatch problem where claims and evidence use different surface forms for the same concept.

Complex claims often require synthesizing information across multiple documents. Multi-hop retrieval chains evidence across sources.

• Iterative retrieval: Uses initial evidence to generate follow-up queries for missing information
• Graph-based traversal: Constructs entity-relationship graphs from retrieved passages and traverses links
• Fusion-in-Decoder: Encodes multiple passages independently but attends across them during decoding
• Chain-of-evidence scoring: Requires that each reasoning step is grounded in a specific retrieved passage

This capability is essential for verifying claims like "Company X acquired Company Y in 2023 for $Z billion," which requires confirming the acquirer, target, date, and amount across sources.

When verifying a claim like "Vitamin D supplementation reduces all-cause mortality," the retrieval system must query PubMed or a proprietary biomedical corpus. The process involves:

• Generating a structured query from the natural language claim
• Searching against MeSH terms and full-text articles
• Ranking results by evidence hierarchy (meta-analyses > randomized controlled trials > observational studies)
• Returning specific excerpts with dosage and outcome data This domain requires high recall to avoid missing critical negative studies.

In the legal domain, evidence retrieval must navigate a corpus of statutes, case law, and regulations. A claim like "The statute of limitations for fraud is 3 years" requires the system to:

• Identify the relevant jurisdiction
• Retrieve the specific statutory text from a legal database
• Find precedential cases that interpret the statute
• Handle temporal reasoning to verify the law is current Dense retrieval models fine-tuned on legal text often outperform generic models due to specialized vocabulary.

For breaking news verification, evidence retrieval cannot rely on static corpora. Systems like ClaimBuster and Google Fact Check Tools continuously index news articles, press releases, and official statements. When a politician claims "Unemployment dropped by 2% this quarter," the system:

• Queries a real-time index of government statistical releases
• Retrieves the official Bureau of Labor Statistics report
• Extracts the specific numerical value for comparison
• Timestamps the evidence to ensure it matches the claim's timeframe

Verifying claims about images requires retrieving evidence from both text and visual corpora. For a claim like "This photo shows a protest in Hong Kong in 2019," the system performs:

• Reverse image search to find prior instances of the image
• Metadata extraction for geolocation and timestamp
• Cross-modal retrieval to find news articles describing the event
• Satellite imagery correlation for environmental verification This combines computer vision embeddings with traditional text retrieval.

Verifying a claim like "CRISPR-Cas9 has off-target effects below 0.1%" requires retrieval beyond simple keyword matching. The system traverses citation graphs to:

• Find the primary research paper making the claim
• Retrieve citing papers that confirm or contradict the finding
• Analyze replication studies for consensus
• Weight evidence by journal impact factor and sample size This graph-based retrieval captures the scientific consensus rather than isolated statements.