Evidence Selection Harness Prompt with JSON Output Contract

This prompt is designed for AI engineers and RAG pipeline builders who need a deterministic, auditable step between retrieval and answer synthesis. The core job-to-be-done is selecting, ranking, and justifying evidence passages before any final text is generated. The ideal user is someone integrating this into a production system where downstream components—such as a synthesis prompt, a citation formatter, or a logging service—consume structured JSON directly. Required context includes a set of retrieved passages, the original user query, and any domain-specific ranking criteria such as source freshness or authority. Without this context, the prompt cannot produce a meaningful selection.

Use this prompt when you need an explicit, machine-readable contract for evidence selection. It is particularly valuable in high-stakes domains like legal research, clinical literature review, or financial analysis, where the reasoning behind including or excluding a source must be auditable. The strict JSON output contract, with fields for rank, score, justification, and exclusion_reason, allows you to build validation and retry logic that fails closed on malformed output. For example, if a downstream validator detects a missing exclusion_reason for a low-scoring passage, the system can trigger a repair loop before the answer is synthesized, preventing unsupported claims from reaching the user.

Do not use this prompt for open-ended chat or when the model should synthesize an answer directly without an intermediate structured selection step. It is also not a replacement for a vector database's native similarity scoring; instead, it adds a layer of semantic reasoning, conflict detection, and explainability on top of initial retrieval results. Before implementing, ensure you have a clear schema for the expected JSON output and a plan for handling malformed responses. The next step is to review the prompt template and adapt the [OUTPUT_SCHEMA] and [CONSTRAINTS] placeholders to match your application's exact contract.

The following prompt template is the core engine for your evidence selection step. It is designed to be called after retrieval, taking a user query and a set of candidate passages, and returning a structured JSON payload that a downstream synthesis prompt or application logic can consume directly. The prompt enforces a strict contract: it scores for relevance and credibility, computes a weighted combined score, filters out low-quality evidence, and provides explicit justifications for every decision. This makes the selection process auditable and debuggable, which is critical for high-stakes applications where hallucination must be minimized.

textCopy
You are an evidence selection engine. Your job is to analyze a set of retrieved passages against a user query and produce a strict JSON output that ranks, scores, and justifies which passages are most relevant and credible.

## Input
- User Query: [USER_QUERY]
- Retrieved Passages: [RETRIEVED_PASSAGES]
- Ranking Criteria: [RANKING_CRITERIA]
- Max Passages to Return: [MAX_PASSAGES]
- Minimum Relevance Score (0.0-1.0): [MIN_SCORE]

## Instructions
1. Score each passage for relevance to the query on a 0.0 to 1.0 scale.
2. Score each passage for credibility based on source authority, recency, and factual consistency on a 0.0 to 1.0 scale.
3. Compute a combined score as (relevance * [RELEVANCE_WEIGHT]) + (credibility * [CREDIBILITY_WEIGHT]).
4. Exclude passages with combined score below [MIN_SCORE].
5. Rank remaining passages by combined score descending.
6. For each excluded passage, provide a specific reason.
7. For each included passage, provide a one-sentence justification.
8. Return only the top [MAX_PASSAGES] passages.
9. Output must be valid JSON matching the schema below. No other text.

## Output Schema
{
  "query": "string",
  "selected_passages": [
    {
      "passage_id": "string",
      "passage_text": "string",
      "relevance_score": number,
      "credibility_score": number,
      "combined_score": number,
      "justification": "string"
    }
  ],
  "excluded_passages": [
    {
      "passage_id": "string",
      "combined_score": number,
      "exclusion_reason": "string"
    }
  ],
  "selection_summary": "string"
}

## Constraints
- Do not fabricate passage content. Use only the provided text.
- If no passages meet the minimum score, return an empty selected_passages array and explain in selection_summary.
- If the query is ambiguous, note this in selection_summary but still rank based on best interpretation.
- All scores must be numbers, not strings.
- Justifications and exclusion reasons must reference specific passage content.

To adapt this template for your application, you must populate the square-bracket placeholders at runtime. [USER_QUERY] and [RETRIEVED_PASSAGES] are your dynamic inputs. [RANKING_CRITERIA] should be a short string describing what matters for your use case (e.g., 'Prioritize recent documentation and official sources'). [RELEVANCE_WEIGHT] and [CREDIBILITY_WEIGHT] are floats that sum to 1.0; start with 0.7 and 0.3 respectively and tune based on your eval results. [MAX_PASSAGES] should be set based on the context window you have budgeted for the subsequent synthesis step. The [MIN_SCORE] acts as a quality gate—set it too low and you risk including noise; set it too high and you may get empty results. Begin with 0.6 and adjust by observing the excluded_passages array in your logs.

Before deploying, you must implement a validation layer around the model's response. The primary failure mode is malformed JSON or missing keys. Your application code should parse the response, validate it against the schema, and implement a retry loop. On a ValidationError, feed the raw output and the error message back to the model with a repair prompt. A secondary failure mode is hallucinated content within the passage_text field; your validator should check that the returned text is an exact substring match of the input passages. If your use case is high-risk—such as legal, financial, or clinical applications—log every response for human audit and consider routing outputs with an empty selected_passages array or a high exclusion rate for mandatory human review before any answer is shown to the user.

This prompt is designed to be called as a pre-synthesis step in your RAG pipeline, after retrieval but before answer generation. Its job is to produce a structured, machine-readable ranking of retrieved passages that downstream synthesis prompts can consume directly. The JSON output contract eliminates the ambiguity of free-text rankings and enables automated validation before evidence reaches the answer-generation stage. Treat this prompt as a deterministic component in your pipeline: it expects a specific input shape, returns a specific output shape, and must be validated before its results are trusted.

Wire the prompt into your application by calling the LLM with the populated template and parsing the response as JSON immediately. Validate the output against the expected schema before passing it downstream. If validation fails, retry up to [RETRY_LIMIT] times (start with 3) by appending the validation error message to the prompt as additional context. This error-feedback loop often corrects malformed JSON, missing fields, or out-of-range scores. If retries are exhausted, log the full failure payload and either fall back to a simpler ranking method (e.g., vector similarity score ordering) or escalate for human review. For model selection, prefer gpt-4o, claude-3-opus, or equivalent models with strong JSON-mode reliability. When using open-weight models, enable any available JSON mode flag and increase your retry budget to 5–7 attempts, as schema adherence is less consistent.

Observability is non-negotiable. Log every call's inputs, outputs, scores, validation results, and retry count. This trace data is essential for debugging ranking quality issues, detecting prompt drift, and comparing performance across model versions. If your RAG system operates in a regulated or high-stakes domain, add a human review gate when confidence scores fall below a configurable threshold or when the evidence set is flagged as insufficient. The structured output from this prompt makes both automated gating and audit review straightforward. Next, integrate the validated ranking output into your synthesis prompt by passing the top-K passages and their scores as structured context, and configure your eval pipeline to compare answer quality with and without this evidence selection step to measure its impact on hallucination reduction.