Retrieval-Augmented Fine-Tuning (RAFT)

RAFT trains on data where a golden 'oracle' document containing the answer is mixed with 'distractor' documents that are topically similar but irrelevant. The model learns to ignore the noise and cite verbatim sequences from the oracle. This directly simulates the messy reality of legal retrieval, where a search for 'breach of fiduciary duty' might return dozens of conceptually related but factually inapplicable cases.

Unlike standard instruction tuning, RAFT explicitly trains the model to ground its answers in direct quotations from the source text. The training objective penalizes paraphrasing and rewards the extraction of exact, attributable sequences. This is critical for legal applications where citation integrity is paramount and a hallucinated quote from a case constitutes professional malpractice.

RAFT fine-tuning data includes detailed reasoning chains that explicitly reference the oracle document. The model learns to generate a step-by-step analysis that connects a legal question to a specific passage in the source material. This produces an auditable reasoning trail, allowing a legal professional to verify the logical derivation from the cited authority to the final conclusion.

A core failure mode of naive RAG is the model being distracted by irrelevant context. RAFT directly inoculates the model against this. By training on a mixture where only a fraction of the provided context is useful, the model develops a selective attention mechanism. It learns to perform a fine-grained comparison between the query and each document, discarding near-misses that a standard model might erroneously synthesize into a plausible but incorrect answer.

For legal applications, RAFT is applied to a base model already pre-trained on legal corpora. The training data consists of synthetic question-answer pairs generated from case law, where the 'oracle' is the binding precedent and 'distractors' are factually similar but distinguishable cases. This teaches the model the specific legal skill of distinguishing precedent, a core competency for any litigation-focused AI system.

Standard fine-tuning for RAG often presents the model with only the correct context, creating a brittle dependency on perfect retrieval. When deployed, the model encounters imperfect retrieval results and fails. RAFT's key innovation is anticipating retrieval failure during training. It forces the model to learn a discriminative function over the provided context, making it resilient to the top-k noise inherent in any real-world legal search system.

In the RAFT framework, distractor documents are irrelevant texts intentionally mixed into the model's context during training. The model must learn to ignore these non-germane passages—such as unrelated statutes or cases from different jurisdictions—while focusing solely on the oracle document that contains the answer. This simulates the noisy retrieval conditions of real-world legal search, where a retriever often returns partially relevant results. By training on a ratio of oracle-to-distractor documents, the model develops robustness against retrieval noise and avoids being misled by superficially similar but legally irrelevant text.

The oracle document is the single ground-truth passage within the training context that contains the exact information needed to answer a legal question. During RAFT training, the model is taught to identify this document and extract verbatim sequences to construct its answer. This differs from standard supervised fine-tuning, where the model simply memorizes answers. Instead, RAFT forces the model to perform in-context reasoning by locating the oracle among distractors and citing it precisely. In legal applications, the oracle might be a specific statutory subsection or a holding paragraph from a controlling precedent.

RAFT extends standard chain-of-thought prompting by requiring the model to articulate its reasoning process while explicitly ignoring distractor documents. The training data includes reasoning traces that demonstrate how to:

• Identify the legally operative document from a set of candidates
• Explain why certain passages are inapplicable due to jurisdictional or factual mismatches
• Extract the precise holding or statutory language that resolves the query This produces a model that not only answers correctly but also generates auditable reasoning chains that legal professionals can verify against the cited sources.

RAFT employs a contrastive learning objective that explicitly penalizes the model for attending to distractor documents. The loss function is structured to reward the model when it generates answers that cite verbatim text from the oracle document and to penalize generations that incorporate information from distractors. This creates a strong training signal that teaches the model to distinguish between authoritative and non-authoritative sources—a critical capability for legal AI systems that must navigate large corpora of persuasive but non-binding authority alongside controlling precedent.

When applied to legal corpora, RAFT is trained on domain-specific question-answer pairs where the oracle document is a real case or statute and distractors are drawn from the same legal database. Key adaptations include:

• Jurisdictional distractors: Cases from different circuits or states that address similar facts but apply different law
• Temporal distractors: Superseded versions of statutes that are no longer good law
• Citation-aware formatting: Training the model to output answers with standard legal citation formats (Bluebook, vendor-neutral) This specialization ensures the model learns the unique evidentiary hierarchies of legal reasoning.

Standard RAG fine-tuning trains a model to use retrieved documents as context but does not explicitly teach it to handle irrelevant results. RAFT differs in three critical ways:

• Explicit distractor training: Standard approaches assume clean retrieval; RAFT assumes noise
• Verbatim citation requirement: RAFT trains the model to quote the source, not paraphrase, reducing hallucination risk
• Reasoning trace supervision: The model learns to show its work, enabling downstream verification For legal applications where citation integrity is paramount, RAFT provides a more robust training paradigm than naive retrieval-augmented instruction tuning.