Evidence Ranking with Citation Metadata Prompt

This prompt is designed for RAG system builders who need a reliable, auditable bridge between retrieval and answer synthesis. Its primary job is to transform a raw, unordered set of retrieved passages into a ranked list, where each entry is paired with a pre-formatted, machine-readable citation metadata block. This is not a general-purpose ranking prompt; it is specifically for pipelines where downstream components require structured source attribution—including title, date, section, and a unique identifier—to generate cited answers or maintain audit logs. The ideal user is an AI engineer or backend developer integrating this step into a production application, where the output must be deterministic enough to be parsed by code and informative enough to be logged for debugging and compliance.

You should use this prompt when you already have a set of retrieved passages and their associated metadata from a search or vector database step. The prompt assumes the retrieval quality is a given; it does not re-execute queries, expand search terms, or fetch additional documents. It is most effective when the downstream answer generation prompt has a strict contract requiring pre-formatted citations, or when you need to log exactly which evidence was selected and why before an answer is generated. A concrete implementation would place this prompt after a retrieval and filtering step but before the final LLM call that synthesizes an answer. For example, a support copilot might retrieve 20 chunks from a knowledge base, use this prompt to rank the top 5 and attach citation JSON, and then inject that ranked JSON directly into the synthesis prompt's [CONTEXT] variable.

Do not use this prompt as a substitute for retrieval, deduplication, or answer generation. It will not correct for poor initial retrieval; if the input passages are irrelevant, the output will be a well-formatted list of irrelevant passages. It is also not designed for conversational state management or multi-turn context tracking—use a dedicated conversational RAG ranking prompt for those scenarios. In high-stakes domains like healthcare or legal, the metadata formatting is necessary but not sufficient; you must still implement human review and source-grounding checks on the final synthesized answer. The next step after reading this section should be to review the prompt template and its required input variables to ensure your retrieval pipeline can supply the necessary metadata fields.

This section provides a complete, copy-ready prompt template for evidence ranking with citation metadata. The template is designed to be pasted directly into your system prompt or user message, with square-bracket placeholders that you replace with your actual data before sending to the model. It produces a ranked list of passages, each accompanied by pre-formatted citation metadata including source title, date, section identifier, and a unique reference ID. This structured output is designed for direct consumption by downstream answer synthesis prompts that need to generate inline citations or footnotes without additional metadata extraction steps.

codeCopy
You are an evidence ranking specialist. Your task is to rank the provided passages by their relevance and credibility for answering the user's question, and to format each passage with complete citation metadata.

## INPUT

**User Question:** [USER_QUESTION]

**Retrieved Passages:**
[RETRIEVED_PASSAGES]

**Source Metadata:**
[SOURCE_METADATA]

## CONSTRAINTS

[CONSTRAINTS]

## OUTPUT SCHEMA

Return a JSON object with the following structure:

{
  "ranked_passages": [
    {
      "rank": 1,
      "passage_id": "string",
      "passage_text": "string",
      "relevance_score": 0.0,
      "relevance_justification": "string",
      "citation_metadata": {
        "source_title": "string",
        "source_date": "string",
        "source_section": "string",
        "source_id": "string",
        "page_or_chapter": "string"
      },
      "exclusion_reason": null
    }
  ],
  "excluded_passages": [
    {
      "passage_id": "string",
      "exclusion_reason": "string"
    }
  ],
  "ranking_summary": "string"
}

## INSTRUCTIONS

1. Evaluate each passage for relevance to the user's question, factual density, and source credibility.
2. Assign a relevance score from 0.0 (completely irrelevant) to 1.0 (perfectly relevant).
3. Rank passages in descending order of relevance.
4. For each ranked passage, extract and format complete citation metadata from the provided source metadata.
5. If a passage is excluded, provide a clear exclusion reason (e.g., "duplicate content", "irrelevant topic", "outdated information", "insufficient factual content").
6. If multiple passages are equally relevant, prioritize the one with higher source credibility or more recent publication date.
7. Include a brief ranking summary explaining your overall approach and any notable decisions.

## EXAMPLES

[EXAMPLES]

## RISK LEVEL

[RISK_LEVEL]

Adaptation guidance: Replace [USER_QUESTION] with the actual query string. [RETRIEVED_PASSAGES] should contain the full text of each retrieved chunk with a unique passage_id for traceability. [SOURCE_METADATA] must include a mapping from passage IDs to their source attributes—title, publication date, section heading, and a durable identifier like a document UUID or URL. The [CONSTRAINTS] placeholder lets you inject domain-specific rules such as "exclude passages older than 2 years" or "require at least 3 unique sources." Use [EXAMPLES] to provide one or two few-shot demonstrations of correct ranking and metadata formatting for your domain. Set [RISK_LEVEL] to low, medium, or high to control the model's conservatism—high-risk domains should trigger stricter exclusion criteria and lower confidence thresholds.

Before deploying this prompt, validate that your source metadata is complete and consistent. Missing metadata fields will cause downstream citation failures. Test the prompt with a golden dataset of known query-passage pairs where you have ground-truth relevance judgments and expected citation formats. Compare the model's output against your expected rankings using nDCG or precision-at-k metrics, and verify that every ranked passage includes all required citation fields with non-null values. For high-stakes applications, add a post-processing validation step that rejects outputs with incomplete metadata and triggers a retry with a more explicit instruction about required fields.

This prompt is designed to be called after retrieval and before answer synthesis in a RAG pipeline. It expects a list of candidate passages and a user query, and it returns a ranked list with structured citation metadata. The output is machine-readable JSON, which means you should validate it before passing it to the next stage. A common integration pattern is: retrieval service → this ranking prompt → top-K selection → answer synthesis prompt. The ranking prompt can also serve as a logging checkpoint for observability into why certain passages were prioritized or dropped.

Wire the prompt into your application with a thin service wrapper that handles the API call, response parsing, and validation. Use a JSON schema validator to confirm that every ranked passage includes the required fields: rank, passage_id, relevance_score, citation_metadata (with source_title, date, section, and unique_id), and ranking_justification. If validation fails, implement a single retry with the error message injected into the prompt as additional context. For high-stakes domains, add a human review step when the top-ranked passage has a relevance score below a configurable threshold (e.g., 0.6) or when the ranking justification indicates source conflict. Log the full ranked list, the selected top-K, and any validation failures for downstream debugging and eval comparison.

Model choice matters here. Use a model with strong instruction-following and JSON output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid smaller or older models that may drop required metadata fields or produce inconsistent ranking justifications. If you are using a local or open-weight model, run a calibration eval comparing its relevance scores against a human-labeled golden set before deploying. Do not pass the raw ranked output directly to users; it is an intermediate representation meant for programmatic consumption. The next step in your pipeline should select the top-K passages (where K is determined by your context window budget) and feed them into your answer synthesis prompt along with the pre-formatted citation metadata.