Fusion-in-Decoder (FiD)

Each retrieved document passage is processed independently through the encoder, producing a set of hidden representations for every passage. This avoids the quadratic complexity of concatenating all passages into a single input sequence. The encoder uses a standard T5 architecture and processes passages in parallel, scaling linearly with the number of retrieved documents.

• Encoder input: question: token followed by passage tokens
• Each passage produces its own set of encoder hidden states
• Passages do not attend to each other during encoding
• Enables scaling to 100+ retrieved passages efficiently

The decoder performs cross-attention over the concatenated hidden states from all encoded passages simultaneously. This is the 'fusion' step—the decoder can attend to tokens across all retrieved documents when generating each output token, enabling it to synthesize evidence from multiple sources.

• Decoder cross-attention spans the union of all passage representations
• Enables multi-hop reasoning across disparate documents
• No passage-level bottleneck; all evidence is accessible at every generation step
• Contrasts with extract-then-read pipelines that isolate passages

FiD's computational cost scales linearly with the number of retrieved passages (O(n)) rather than quadratically (O(n²)). This is because self-attention is confined to individual passages in the encoder, while the decoder's cross-attention cost grows linearly with the total token count across all passages.

• Encoder self-attention: bounded by passage length, not corpus size
• Decoder cross-attention: O(total_tokens) across all passages
• Practical for n=100 passages at standard sequence lengths
• Enables retrieval-augmented generation at scale without prohibitive compute

FiD is built on the T5 (Text-to-Text Transfer Transformer) encoder-decoder architecture. The text-to-text framing means both the input (question + passages) and output (answer) are treated as text strings, enabling a unified training objective: maximum likelihood estimation over the target answer tokens.

• Initialized from pre-trained T5 weights (e.g., T5-base, T5-large)
• Fine-tuned on knowledge-intensive tasks like open-domain QA
• Output is generated autoregressively, token by token
• Leverages T5's strong transfer learning capabilities

Unlike extractive QA systems that must first identify a single answer span within a specific passage, FiD performs abstractive generation by fusing information from all passages in the decoder. This allows the model to synthesize answers that are not verbatim present in any single retrieved document.

• Generates answers that may combine facts from multiple passages
• Handles conflicting evidence through learned weighting in cross-attention
• No hard commitment to a single source passage
• Produces more coherent, natural-language answers than extractive methods

FiD is trained end-to-end using supervised learning on question-answer pairs where the input includes retrieved passages from a dense retriever (typically DPR). The model learns to ignore irrelevant passages and focus cross-attention on evidence-bearing tokens, developing robustness to retrieval noise.

• Training data: (question, retrieved_passages, ground_truth_answer) triples
• Retriever is typically frozen; FiD learns to leverage its outputs
• Loss: standard cross-entropy over generated answer tokens
• Naturally handles distractor passages without explicit noise modeling

FiD is the foundational architecture for answering complex, fact-seeking queries against a corpus of millions of documents, such as Wikipedia.

• Mechanism: A retriever fetches 100+ relevant passages; the encoder processes each independently; the decoder performs cross-attention over all passages simultaneously to generate a single answer.
• Example: Answering "What is the primary function of the ribosome in a eukaryotic cell?" requires synthesizing evidence from multiple biology articles.
• Key Advantage: Unlike RAG, which prepends text to the prompt, FiD allows the decoder to jointly reason over all evidence, mitigating the Lost-in-the-Middle problem.

FiD excels at generating a concise, fluent summary from a large collection of documents on a single topic, such as news articles or financial reports.

• Workflow: The encoder processes each document chunk independently, and the decoder generates a summary that aggregates key points, resolves contradictions, and eliminates redundancy.
• Enterprise Use Case: Automatically generating an executive briefing from a day's worth of market reports, press releases, and analyst notes.
• Technical Benefit: The encoder-decoder structure provides more generative control than extractive methods, producing abstractive summaries that are more coherent.

FiD powers chatbots that must answer user questions by grounding every claim in a specific, retrievable document from a knowledge base.

• Architecture: User query and dialogue history are used to retrieve relevant support articles. FiD fuses these articles to generate a response with citation integrity.
• Example: A customer support bot for a software company that answers technical questions by synthesizing steps from multiple documentation pages.
• Why FiD: It reduces hallucination by forcing the decoder to attend to all provided evidence, making it ideal for regulated industries like finance and healthcare.

FiD is used to verify the truthfulness of a claim by aggregating evidence from a corpus of trusted sources and generating a verdict.

• Process: A claim is input, a retriever finds supporting and refuting evidence, and FiD's decoder generates a label (e.g., SUPPORTS, REFUTES, NOT ENOUGH INFO) along with a rationale.
• Application: Automated fact-checking pipelines for newsrooms or social media platforms to combat misinformation at scale.
• Architectural Fit: The model's ability to weigh conflicting evidence from multiple passages in the decoder is critical for nuanced verdicts.

In biomedicine, FiD synthesizes answers to clinical questions from a corpus of research abstracts and trial reports, such as PubMed.

• Task: Given a PICO (Population, Intervention, Comparison, Outcome) query, retrieve relevant studies and generate a summary of clinical efficacy.
• Impact: Dramatically reduces the time clinicians spend on systematic literature reviews by providing an initial, evidence-grounded synthesis.
• Domain Adaptation: FiD models are often fine-tuned on biomedical corpora to handle specialized terminology and complex entity relationships.

FiD processes large volumes of case law and contracts to answer complex legal questions that require reasoning over multiple precedents.

• Mechanism: A query about a legal principle retrieves hundreds of relevant case snippets. FiD's decoder performs joint reasoning to identify the controlling precedent and generate a synthesized answer.
• Use Case: A legal research tool that answers "Under what circumstances is a liquidated damages clause considered an unenforceable penalty?" by analyzing dozens of cases.
• Key Requirement: The architecture's ability to handle long-range dependencies across documents is essential for tracing legal reasoning chains.

The core architectural component that enables FiD's decoder to perform joint reasoning. In the Transformer decoder, cross-attention allows every generated token to attend to the hidden representations of all tokens from all retrieved passages simultaneously. This is the mechanism that fuses information from multiple independent sources, distinguishing it from encoder-only fusion methods.

FiD is fundamentally built on a sequence-to-sequence encoder-decoder framework, typically T5. The encoder processes each retrieved passage independently, generating a set of hidden representations. The decoder then uses cross-attention to aggregate these representations. This separation is critical for FiD's scalability, as encoder computation is linear with the number of passages.

A critical optimization step where a more precise, computationally intensive model re-scores the initial set of retrieved documents before they enter the FiD encoder. Effective re-ranking ensures that the most semantically relevant passages occupy the limited input slots, maximizing the quality of the decoder's joint reasoning and preventing the dilution of the attention mechanism with noisy text.

The class of problems FiD was designed to solve. These tasks require access to a large corpus of external world knowledge that cannot be stored in a model's parameters. Examples include:

• Open-domain Question Answering: Answering factoid questions without a specified context.
• Fact Verification: Checking the veracity of a claim against a knowledge source.
• Dialogue Generation: Producing knowledgeable responses grounded in external documents.