Evidence Strength Rating Prompt Template

This prompt is designed for RAG evaluators and retrieval pipeline engineers who need to programmatically assess the quality of individual retrieved passages before they are used for answer generation. The core job-to-be-done is to produce a structured, per-source quality signal that can be used for evidence ranking, retrieval quality monitoring, or as a filtering gate in a production RAG system. The ideal user is an AI engineer or MLOps specialist building an automated evaluation harness, not an end-user chatting with a model. They need a deterministic, repeatable rating that can be logged, compared across pipeline versions, and used to trigger re-retrieval or escalation.

Use this prompt when you have a list of retrieved passages and need to rank them by composite strength before synthesis. It is appropriate for offline evaluation of retrieval systems, online re-ranking in a RAG pipeline, and for generating quality metrics for dashboards. Do not use this prompt as a standalone answer generator; it rates evidence, it does not answer the user's question. It is also not a replacement for a full factuality check—it assesses surface-level quality signals (relevance, authority, specificity, recency) but does not verify claims against a ground-truth knowledge base. If your system requires claim-level verification, pair this with a dedicated fact-checking prompt from the Fact Checking and Claims Verification pillar.

The prompt requires the following inputs: the user's original question, the retrieved passage text, and metadata about the source (publication date, author, domain). It outputs a structured JSON object with individual dimension scores and a composite strength score. Before deploying, you must calibrate the scoring rubric against human judgments on your specific domain data. Generic scoring heuristics will drift across legal, medical, and technical domains. Start by running this prompt against a golden dataset of 50–100 passage-question pairs with known quality labels, and tune the rubric descriptions in the prompt until the model's scores correlate with your ground truth. Once calibrated, wire the output into your evidence ranking step and log every score for regression testing when you change retrieval parameters.

The Evidence Strength Rating prompt is designed to be dropped into a RAG evaluation pipeline or a pre-generation evidence ranking step. It forces the model to assess each retrieved passage against four explicit dimensions—relevance, authority, specificity, and recency—before producing a composite score. This structured approach prevents the model from defaulting to a vague 'this looks good' heuristic and instead produces auditable, per-dimension ratings that you can log, aggregate, and use to tune retrieval quality over time.

Below is the copy-ready template. Replace each square-bracket placeholder with your application's actual values before sending the request. The [PASSAGES] placeholder expects a list of retrieved chunks with their metadata. The [QUESTION] placeholder holds the user's original query. The [OUTPUT_SCHEMA] placeholder should contain your expected JSON structure, and [CONSTRAINTS] is where you define scoring rules, dimension definitions, and any domain-specific weighting. For high-stakes domains such as clinical or legal review, add a [RISK_LEVEL] parameter that triggers stricter recency and authority thresholds.

textCopy
You are an evidence quality evaluator for a retrieval-augmented generation system. Your job is to rate each retrieved passage on four dimensions and produce a composite strength score.

## Input
- Question: [QUESTION]
- Retrieved Passages: [PASSAGES]

## Rating Dimensions
For each passage, assign a score from 1 (lowest) to 5 (highest) on each dimension:

1. **Relevance**: How directly does this passage address the question? (1 = unrelated, 5 = directly answers)
2. **Authority**: How credible is the source? Consider publication venue, author expertise, and institutional backing. (1 = anonymous/unverified, 5 = authoritative primary source)
3. **Specificity**: How precise and detailed is the information? (1 = vague generalities, 5 = specific facts, figures, or procedures)
4. **Recency**: How current is the information relative to the question's temporal needs? (1 = clearly outdated, 5 = recent and timely)

## Scoring Rules
- If a passage is completely irrelevant (Relevance = 1), set all other dimensions to 1 and Composite = 1.
- If authority cannot be determined from available metadata, default to 3 and note 'authority_unknown'.
- If the question has no temporal dependency, score Recency as 3 and note 'recency_not_applicable'.
- [CONSTRAINTS]

## Output Format
Return a JSON object matching this schema exactly:
[OUTPUT_SCHEMA]

## Example Output
```json
{
  "question": "What are the side effects of drug X?",
  "passage_ratings": [
    {
      "passage_id": "doc_17_chunk_3",
      "relevance": 5,
      "authority": 4,
      "specificity": 4,
      "recency": 5,
      "composite": 4.5,
      "notes": "Peer-reviewed journal article from 2024. Lists specific side effects with incidence rates."
    }
  ],
  "rating_summary": {
    "average_composite": 4.5,
    "highest_rated_passage": "doc_17_chunk_3",
    "lowest_rated_passage": null,
    "passages_below_threshold": []
  }
}

After copying the template, the most important adaptation step is defining your [CONSTRAINTS]. This is where you encode domain-specific rules that prevent the model from overrating weak evidence. For example, in a financial compliance setting, you might add: 'If the passage is from a social media post or unverified forum, cap Authority at 2.' In a medical context: 'If the passage references a study older than 5 years without a confirmatory recent citation, cap Recency at 2.' These constraints turn a generic rating prompt into a domain-calibrated quality gate. Always test constraint wording against a golden set of passages where you know the correct ratings—vague constraints produce inconsistent scores, while overly strict constraints can cause the model to refuse to rate legitimate evidence. Start with 10-20 labeled examples, run the prompt, and compare per-dimension scores to your ground truth before deploying to production.