Evidence Ranking with Rubric Prompt Template

This prompt is for evaluation engineers and AI quality teams who need to move beyond binary relevance judgments when ranking evidence. The core job-to-be-done is calibrating a model to score multiple evidence passages against a defined, multi-dimensional rubric—factoring in dimensions like factual precision, query alignment, and source authority—to produce a ranked list suitable for gating downstream answer generation. The ideal user is someone integrating a ranking step into a RAG pipeline or an automated evaluation harness, where the output must be a structured, scored, and justified ranking rather than a simple ordered list.

Use this prompt when you have a retrieval set that is too large or noisy for direct answer generation, and you need a transparent, auditable method for selecting the strongest evidence. It is particularly effective when your application requires calibrated confidence scores to decide whether to generate an answer, abstain, or escalate for human review. The required context includes the user's query, the set of evidence passages to rank, and a pre-defined scoring rubric with clear dimension descriptions and weightings. Do not use this prompt for simple binary relevance filtering, for real-time ranking where latency is measured in milliseconds, or when the evidence set is so small that a rubric adds unnecessary overhead.

The primary constraint is that the rubric itself must be well-defined before this prompt is used. A vague rubric produces uncalibrated scores that are worse than no ranking at all. Before implementing this prompt, invest time in defining what 'factual precision' or 'source authority' means for your domain, and test the rubric against a golden set of ranked passages to ensure inter-rater consistency. The next step after reading this section is to review the prompt template and adapt the rubric dimensions to match your specific evidence quality standards.

This template is designed to be dropped directly into your evidence ranking pipeline. It expects a list of evidence passages and a defined scoring rubric with dimensions such as factual precision, query alignment, source authority, and specificity. The model ranks each passage against every rubric dimension, assigns composite scores, and provides concise rationale for each ranking decision. Use this when you need calibrated, explainable evidence rankings that can gate downstream answer generation—not just a flat relevance sort.

textCopy
You are an evidence ranking specialist. Your job is to evaluate each evidence passage against a defined scoring rubric and produce a ranked list with calibrated scores and rationale.

## INPUTS

### Query
[QUERY]

### Evidence Passages
[EVIDENCE_PASSAGES]

### Scoring Rubric
[RUBRIC_DIMENSIONS]

### Output Schema
[OUTPUT_SCHEMA]

### Constraints
[CONSTRAINTS]

## INSTRUCTIONS

1. For each evidence passage, evaluate it against every rubric dimension independently.
2. Assign a score for each dimension on the scale defined in the rubric. Do not inflate scores when evidence is weak.
3. Compute a composite score using the weighting specified in the rubric. If no weights are provided, weight all dimensions equally.
4. Rank passages from highest to lowest composite score.
5. For each passage, provide a one-sentence rationale explaining the most important factor that determined its rank.
6. If a passage is too vague, off-topic, or contradicts the query's core claim, flag it as [WEAK_EVIDENCE] and place it at the bottom of the ranking with a score of 0.
7. If two passages are near-duplicates, select the more specific or authoritative one and note the duplication in the rationale.
8. If no passage meets the minimum threshold defined in [CONSTRAINTS], return an empty ranking with a note that evidence is insufficient.

## OUTPUT FORMAT

Return valid JSON matching the [OUTPUT_SCHEMA] exactly. Do not include commentary outside the JSON structure.

## EXAMPLE

Query: "What is the capital of France?"

Evidence Passages:
- Passage A: "Paris is the capital and largest city of France."
- Passage B: "France is a country in Western Europe."

Rubric Dimensions:
- Factual Precision (0-10): Does the passage contain a directly stated fact that answers the query?
- Query Alignment (0-10): How closely does the passage address the specific question asked?
- Source Authority (0-10): How authoritative is the source? (Assume equal authority if not specified.)

Output:
{
  "ranked_passages": [
    {
      "passage_id": "A",
      "scores": {
        "factual_precision": 10,
        "query_alignment": 10,
        "source_authority": 5
      },
      "composite_score": 8.33,
      "rationale": "Directly states the answer with high precision and perfect query alignment."
    },
    {
      "passage_id": "B",
      "scores": {
        "factual_precision": 0,
        "query_alignment": 2,
        "source_authority": 5
      },
      "composite_score": 2.33,
      "rationale": "Provides context about France but does not answer the specific question about its capital."
    }
  ],
  "insufficient_evidence": false
}

Adaptation guidance: Replace [RUBRIC_DIMENSIONS] with your actual scoring dimensions, scales, and weights. Define [OUTPUT_SCHEMA] as a strict JSON schema that your application validator expects—include field types, required fields, and enum constraints. Set [CONSTRAINTS] to specify minimum score thresholds, maximum passage count, and any domain-specific rules (e.g., "prefer peer-reviewed sources over blog posts"). The [EVIDENCE_PASSAGES] placeholder should receive passages with unique IDs already assigned by your retrieval system. If your use case is high-risk—such as medical, legal, or financial ranking—add a human review step before ranked evidence feeds into answer generation. Test this prompt against known ranking failure modes: position bias (first-passage preference), length bias (longer passages scoring higher), and authority overfitting (ignoring content quality when source metadata is strong). Run the output through a JSON schema validator and retry with error context if validation fails.

This prompt is designed to sit between your retrieval system and your answer generation step. It takes a set of retrieved passages and a defined scoring rubric, then returns calibrated scores and rankings. The implementation harness must treat this prompt as a deterministic quality gate, not a best-effort suggestion. If the top-ranked passage falls below a minimum score threshold, the system should refuse to generate an answer and instead request better evidence or inform the user that the query cannot be grounded.

Wire the prompt into your application as a synchronous pre-generation step. After retrieval returns N passages, assemble the prompt with the rubric dimensions, the query, and the passage set. Send the request to a model capable of structured output (e.g., GPT-4o with response_format or Claude with tool use for JSON). Validate the output before proceeding: confirm that every passage ID in the ranking exists in the input set, that scores fall within the defined rubric range, and that the ranking is non-empty. If validation fails, retry once with a stricter output contract. If the retry also fails, log the failure, fall back to a simpler relevance-only ranking, and flag the incident for review.

For high-stakes domains such as healthcare, legal, or finance, insert a human review step when the top-ranked evidence score is below the HIGH confidence tier or when multiple passages receive identical top scores. The harness should surface the ranked list, the rubric scores, and the original passages to a review queue. Do not auto-generate an answer from low-confidence evidence. Additionally, log every ranking request and response—including the prompt version, model, rubric used, scores, and any validation failures—so that ranking quality can be audited and regressions detected across prompt changes.

Model choice matters here. Use a model with strong instruction-following and structured output support. Avoid models that exhibit position bias (favoring first or last passages) without mitigation; randomize passage order in the prompt to reduce this effect. If your latency budget is tight, consider running this prompt on a smaller, fine-tuned model that has been trained on your rubric and ranking preferences. The rubric itself should be version-controlled alongside the prompt template, and any change to the rubric dimensions or scoring scale should trigger a regression test against a golden ranking dataset before deployment.