FLARE Retrieval

FLARE monitors the model's log probability for each generated token. When the probability of an upcoming token drops below a defined confidence threshold, the system interprets this as a knowledge gap. Instead of hallucinating, it pauses generation and formulates a search query from the preceding context to retrieve missing legal authority.

FLARE does not require a user to ask a question. It constructs a search query automatically from the incomplete sentence it was about to generate. This implicit query captures the latent information need—often a specific case name, statutory citation, or legal principle—that the model requires to continue its reasoning chain with high confidence.

FLARE operates in a retrieval-interleaved generation loop. The process alternates between:

• Generate: Produce tokens until confidence drops.
• Retrieve: Query the legal corpus with the pending sentence.
• Re-Generate: Continue generation with the retrieved documents appended to the context. This cycle repeats until the full legal analysis is complete.

By design, FLARE prevents the model from generating low-probability tokens that often correspond to factual fabrications. In legal AI, this is critical for avoiding phantom citations or misstated holdings. The architecture ensures that every propositional step is grounded in retrieved text before it is committed to the output.

FLARE is corpus-agnostic but particularly effective when paired with a high-recall legal retrieval system. It relies on the underlying retriever to return relevant documents from a jurisdictionally filtered index. The quality of the proactive retrieval is directly dependent on the semantic density and authority scoring of the legal embedding space.

Unlike standard RAG, which performs a single retrieval upfront based on the user's question, FLARE performs multiple, dynamic retrievals mid-generation. This allows it to handle complex legal reasoning that requires connecting facts from one case to the holding of another—a process that cannot be fully anticipated by a single initial query.

The core mechanism that triggers FLARE. The system monitors the log probability of each generated token. When the model's confidence drops below a calibrated threshold—indicating impending uncertainty or hallucination—the generation is paused. The low-probability sentence fragment is converted into a search query to retrieve grounding documents before resuming generation.

FLARE is a specific implementation of this broader decoding strategy. Instead of retrieving all context upfront, the model alternates between generating reasoning and issuing new queries. This allows the system to actively seek evidence for each logical step, mimicking how a lawyer might research a novel point mid-argument.

Standard RAG performs a single retrieval step before generation. FLARE's active retrieval differs by iterating. Key distinctions:

• Single-Pass: Retrieves once based on the user query.
• Active (FLARE): Retrieves multiple times based on the model's own generation trajectory. This makes FLARE particularly suited for multi-hop legal reasoning where the necessary evidence chain is unknown a priori.

When confidence drops, FLARE must convert the tentative, low-probability text into a viable search query. This involves masking uncertain tokens and using the surrounding high-confidence context to formulate a query. For legal applications, this often means extracting a nascent legal proposition and converting it into a Boolean or natural language query against a case law database.

Two primary methods signal the need for retrieval:

• Implicit FLARE: Uses the model's internal token probability as a direct confidence signal. A drop below a threshold (e.g., < 0.5) triggers retrieval.
• Explicit FLARE: Instructs the model to generate a special retrieval token (e.g., [Search(term)]) when it needs information, leveraging the model's own metacognitive assessment of its knowledge gaps.

For multi-document legal reasoning, FLARE must be paired with a domain-specific retrieval backend. The proactive queries generated by the model are executed against a legal search index (e.g., a hybrid legal search system). The retrieved precedents or statutes are then injected into the context window, allowing the model to ground its low-confidence legal proposition in authoritative text before continuing.