Refusal-to-Answer Prompt for Legal RAG Systems

This prompt is for legal tech teams deploying Retrieval-Augmented Generation (RAG) systems where a wrong answer creates direct liability, not just a poor user experience. The job-to-be-done is a safe, jurisdiction-aware refusal that protects the deploying organization while still providing utility through alternative resources. The ideal user is an AI engineer or product manager who needs a deterministic gate that prevents the model from inventing case law, misinterpreting statutes, or offering unauthorized legal advice. This is not a general-purpose safety prompt; it is a domain-specific instrument for high-stakes legal Q&A where the cost of hallucination outweighs the cost of silence.

Use this prompt when your RAG system faces queries that fall outside retrieved context, ask for legal predictions, or cross jurisdictional boundaries where the system lacks authoritative sources. It is appropriate when you have already implemented retrieval, evidence ranking, and source sufficiency checks, and you need a final refusal layer that produces a structured, auditable output. Do not use this prompt as a replacement for retrieval quality or evidence grounding. If your RAG pipeline regularly retrieves irrelevant documents, fix retrieval first. This prompt is also inappropriate for low-risk legal FAQs, general legal information sites, or internal research tools where a qualified legal professional reviews every output before use.

Before implementing, confirm you have the required inputs: a user query, retrieved context with source metadata, and a defined jurisdiction scope. The prompt expects these as [QUERY], [RETRIEVED_CONTEXT], and [JURISDICTION_SCOPE] placeholders. You must also define your refusal policy boundaries—what question types trigger refusal versus a caveated answer. After deployment, monitor both over-refusal and under-refusal rates against a golden dataset of borderline queries. A refusal prompt that blocks every question is safe but useless; one that answers everything is dangerous. The next section provides the copy-ready template you can adapt to your specific legal domain and risk tolerance.

This template is designed to be the final generation step in a legal RAG pipeline. It assumes that a prior retrieval step has returned context, and that an upstream abstention or confidence gate has already determined that the system should refuse to answer. The prompt's job is to produce a refusal that is legally safe, jurisdiction-aware, and helpful rather than evasive. It does not decide whether to refuse—it executes the refusal with the right structure and tone.

textCopy
SYSTEM:
You are a legal research assistant operating under strict liability constraints. Your primary directive is to refuse to answer when the available legal context is insufficient, ambiguous, or outside your authorized scope. You must never fabricate case law, statutes, or legal conclusions.

When refusing to answer, you MUST:
- State clearly that you cannot answer the question.
- Explain the specific reason for refusal using one of the defined reason codes.
- Include a jurisdiction disclaimer if the question implies a specific jurisdiction.
- Suggest at least one alternative resource or next step the user can take.
- Never speculate about what the law might be.
- Never provide partial legal analysis that could be misinterpreted as advice.

Reason codes:
- INSUFFICIENT_CONTEXT: Retrieved documents do not contain enough information.
- OUT_OF_SCOPE: Question requires legal advice, strategy, or prediction.
- JURISDICTION_MISMATCH: Available sources do not cover the implied jurisdiction.
- CONFLICTING_SOURCES: Retrieved authorities contradict each other and resolution requires legal judgment.
- AMBIGUOUS_QUERY: Question is too vague to ground in specific legal authority.

USER:
Query: [QUERY]

Retrieved Context:
[CONTEXT]

Jurisdiction (if known): [JURISDICTION]

Risk Level: [RISK_LEVEL]

Output Format:
{
  "answer": "I cannot answer this question.",
  "refusal": {
    "reason_code": "[one of the defined codes]",
    "explanation": "[specific, non-speculative explanation grounded in the retrieved context or query characteristics]",
    "jurisdiction_disclaimer": "[jurisdiction-specific disclaimer or 'No jurisdiction assumed' if unknown]",
    "alternative_resources": ["[actionable next step 1]", "[actionable next step 2]"]
  },
  "citations": []
}

Constraints:
[CONSTRAINTS]

To adapt this template, replace each square-bracket placeholder with values from your application layer. [QUERY] should carry the user's original question. [CONTEXT] should contain the retrieved passages that triggered the refusal decision—include them so the model can explain why they were insufficient. [JURISDICTION] can be extracted from the query or set to an empty string. [RISK_LEVEL] should be set by your upstream classifier as LOW, MEDIUM, or HIGH to adjust the tone of the disclaimer. [CONSTRAINTS] is optional and can carry additional policy rules such as 'Never mention specific law firm names' or 'Always include the state bar referral service.' The output schema is intentionally rigid: the citations array must remain empty during a refusal to prevent hallucinated sources. Validate this field in your application layer and trigger a retry or escalation if citations appear in a refusal response.

Integrating a refusal-to-answer prompt into a production legal RAG system requires treating the prompt as one component inside a larger application harness. The harness is responsible for assembling the prompt with the correct [CONTEXT] (retrieved passages), [QUERY] (user question), and [JURISDICTION] (configured legal region), then parsing the model's structured output to decide whether to display an answer, a refusal, or escalate for human review. Because legal RAG systems operate in a high-liability domain, the harness must enforce strict validation on the model's output schema, log every decision for auditability, and provide a clear path for human override when the model's abstention logic is uncertain.

A robust implementation begins with a validation layer that parses the model's JSON response and checks for the required fields: decision (must be one of ANSWER, REFUSE, or ESCALATE), reason_code, jurisdiction_disclaimer, and alternative_resources. If the model returns malformed JSON or missing fields, the harness should retry the prompt once with a stronger format reminder appended to the system message. After two consecutive failures, the harness must escalate to a human review queue rather than silently falling back to a generated answer. For model choice, use a model with strong instruction-following and JSON mode support (e.g., gpt-4o or claude-3-5-sonnet with structured outputs enabled). Set temperature=0 to minimize variance in refusal decisions. The retrieval pipeline should supply the top k chunks ranked by relevance, with source metadata (document title, date, jurisdiction tags) included in the [CONTEXT] block so the model can assess authority and recency.

Logging and audit trails are non-negotiable in legal applications. Every invocation should log: the user query, the retrieved context sources, the raw model response, the parsed decision, the reason code, and whether the output was shown to the user or escalated. Store these records in an append-only datastore with retention aligned to your organization's legal document retention policy. For human review integration, when the model returns ESCALATE or when validation fails, the harness should push a review task to a queue (e.g., a legal team's ticketing system or a review UI) containing the original query, retrieved context, and the model's raw output for the reviewer to accept, override, or annotate. The reviewer's decision should be logged back into the audit trail to close the loop. Avoid building a system where the model's refusal is the final word without human recourse; legal queries often sit in gray areas where a domain expert's judgment is required.

Adversarial testing must be part of the deployment harness, not a one-time pre-release check. Maintain a regression suite of boundary-pushing queries: questions that appear answerable but rely on missing facts, questions that mix jurisdictions, questions that ask the model to interpret law rather than summarize it, and questions designed to elicit over-confident answers. Run this suite against every prompt or model version change and compare refusal rates, reason code distributions, and false-answer rates. Set thresholds for acceptable behavior: for example, a false-answer rate above 0.5% on adversarial queries should block deployment. Finally, implement a production monitoring dashboard that tracks refusal rate, escalation rate, validation failure rate, and human review turnaround time. A sudden drop in refusal rate may indicate prompt drift or a model behavior change; a spike in escalations may indicate retrieval quality degradation. Wire alerts to these metrics so the team can investigate before a liability event occurs.