Fusion-in-Decoder (FiD)

Each retrieved passage is encoded separately and in parallel by the encoder, without cross-attention between passages. This means:

• Linear scaling: Processing 100 passages costs O(n) rather than O(n²) in the encoder
• No token limit bottleneck: Each passage can use the full encoder context window independently
• Trivial batching: Passages are processed as independent items in a batch, maximizing GPU utilization

The encoder outputs a set of hidden representations for each passage, which are then concatenated into a single sequence for the decoder.

The decoder performs cross-attention over the concatenated representations of all passages simultaneously. This fusion mechanism:

• Allows the model to compare and contrast evidence from different sources
• Enables evidence synthesis where facts from multiple documents are combined into a single answer
• Preserves the ability to attribute generated text to specific passages through attention weight analysis

The decoder sees all evidence at once, unlike retrieve-and-read architectures that process passages sequentially or truncate to fit a single context window.

FiD's architectural separation of encoding and fusion provides a fundamental scalability advantage over long-context approaches:

• Encoder: Scales linearly with passage count, enabling retrieval of hundreds of documents
• Decoder: Scales quadratically with total concatenated length, but this is manageable because only the decoder pays the O(n²) cost
• Memory efficiency: Encoder activations can be computed once and cached, then reused across multiple decoding steps

This makes FiD particularly effective for open-domain question answering where the retriever returns dozens or hundreds of candidate passages.

FiD models are trained with a supervised sequence-to-sequence objective where:

• The input is a set of retrieved passages concatenated with the question
• The target is the ground-truth answer string
• Teacher forcing is used during training, with the decoder attending to all encoded passages
• The retriever and reader can be trained jointly or separately depending on the implementation

Key training considerations include passage sampling strategy (how many passages, which ones) and negative example construction to teach the model to ignore irrelevant retrieved content.

FiD occupies a distinct position in the RAG architecture landscape:

• vs. Long-Context Models: FiD avoids the O(n²) encoder cost of processing all passages as one long sequence
• vs. Retrieve-and-Read: FiD fuses evidence in the decoder rather than concatenating before encoding, preserving per-passage independence
• vs. Map-Reduce: FiD performs joint reasoning rather than independent summarization followed by aggregation
• vs. Iterative Retrieval: FiD processes all passages in one forward pass rather than sequentially refining retrieval

The trade-off is that FiD requires a fixed retrieval set before generation begins, limiting dynamic retrieval during decoding.

Because the decoder attends to all encoded passages jointly, FiD provides natural interpretability through attention analysis:

• Passage-level attribution: Aggregate attention weights reveal which passages most influenced each generated token
• Cross-passage dependencies: Attention patterns show how the model connects information across documents
• Evidence provenance: Generated claims can be traced back to specific source passages for verification

This transparency is valuable for factual grounding and hallucination detection in production RAG systems, as it provides a built-in mechanism for citation generation.