Inferensys

Prompt

Evidence Ranking for Citation Selection Prompt

A practical prompt playbook for using Evidence Ranking for Citation Selection Prompt in production AI workflows.
Developer doing prompt engineering on laptop, prompt variations visible on screen, casual coding session.
PROMPT PLAYBOOK

When to Use This Prompt

Defines the job-to-be-done, ideal user, required context, and boundaries for the Evidence Ranking for Citation Selection Prompt.

This prompt is for RAG system builders and citation-aware application engineers who need to move beyond topical relevance and identify exactly which retrieved passages contain directly quotable support for a claim. Its single job is to rank a provided retrieval set by citation worthiness: direct quote support, claim alignment, and source verifiability. Use this prompt when your application must produce answers with inline citations, when auditability of source grounding matters, or when you need to filter a large retrieval set down to the passages that actually contain claim-level evidence. The ideal user already has a retrieval pipeline producing candidate passages and needs a reliable ranking step before answer generation.

This prompt assumes you already have a retrieval set. It does not perform retrieval, rewrite queries, or generate the final answer. It does not handle multi-hop reasoning across passages or resolve contradictions between sources—those are separate concerns. The prompt is designed for single-step ranking: given a claim or question and a set of candidate passages, it returns a ranked list with explicit rationale for why each passage is or is not citation-worthy. This makes it suitable as a mid-pipeline component that feeds into a downstream answer generation step with strict citation requirements.

Do not use this prompt when you need raw topical relevance scoring without citation intent, when your retrieval set has not been pre-filtered for basic relevance, or when you need the model to generate the final answer directly. It is also not a replacement for retrieval itself—feeding it hundreds of passages without pre-filtering will produce noisy rankings and waste context budget. For high-stakes domains like legal or medical applications, always pair this prompt with human review of the ranked passages before they are used in a final answer. The ranking is a recommendation, not a verdict. If your application cannot tolerate any hallucinated citations, implement a downstream verification step that checks whether the cited passage actually contains the quoted claim.

PRACTICAL GUARDRAILS

Use Case Fit

Where the Evidence Ranking for Citation Selection Prompt works, where it fails, and the operational preconditions required before you put it in front of users.

01

Good Fit: Citation-Aware RAG Pipelines

Use when: your application must produce answers with inline citations that map claims to specific source spans. The prompt is designed to rank passages by direct quote support and claim alignment, making it ideal for legal-tech, compliance, and research assistants. Guardrail: always verify that ranked passages actually contain the cited spans before surfacing to users.

02

Bad Fit: Open-Ended Summarization

Avoid when: the task is general summarization without citation requirements. This prompt optimizes for claim-to-evidence mapping, not narrative coherence or topic coverage. Using it for abstractive summaries will produce fragmented, quote-heavy output. Guardrail: route summarization tasks to a dedicated synthesis prompt and reserve this for citation selection workflows.

03

Required Inputs: Retrieved Passages with IDs

What to watch: the prompt assumes each passage has a stable identifier and retrievable text span. Without passage IDs, citation mapping breaks. Without sufficient retrieval recall, the ranker has nothing useful to rank. Guardrail: validate that your retrieval step returns at least 5-10 candidate passages with unique IDs before invoking this prompt. Log empty retrieval sets as upstream failures.

04

Operational Risk: Hallucinated Citations

What to watch: the model may select a passage as citation-worthy but fabricate a quote or span reference that does not exist in the source. This is the highest-severity failure mode for citation-aware systems. Guardrail: implement a post-processing verification step that extracts the claimed span from the source document and checks for exact or near-exact match before the citation reaches the user.

05

Operational Risk: Authority Overfitting

What to watch: the prompt may overweight a single high-authority source and ignore equally relevant passages from less authoritative but still credible sources. This produces citation monoculture. Guardrail: add a diversity constraint in the ranking criteria that requires at least two distinct sources in the top-K selections when multiple credible sources exist.

06

Operational Risk: Position Bias in Passage Ordering

What to watch: the model may favor passages that appear first in the input list, especially when passages are similarly relevant. This skews citation selection toward retrieval order rather than evidence quality. Guardrail: randomize passage order before sending to the prompt and run stability checks across multiple orderings in your eval harness.

PROMPT PLAYBOOK

Copy-Ready Prompt Template

A production-ready template for ranking evidence passages by citation strength, claim alignment, and source verifiability.

This template is designed to be pasted directly into your prompt layer. It instructs the model to evaluate a set of retrieved evidence passages against a specific claim or question, ranking them by their suitability for citation in a final answer. The prompt prioritizes direct quote support, alignment with the claim, and the verifiability of the source. All dynamic inputs are represented as square-bracket placeholders, which your application must replace before sending the request to the model.

text
You are an evidence-ranking specialist. Your task is to evaluate a set of retrieved evidence passages and rank them by their suitability for citation in a final answer to a user's question.

# INPUT

**User Question:** [USER_QUESTION]

**Claim to Support:** [CLAIM_TO_SUPPORT]

**Retrieved Evidence Passages:**
[EVIDENCE_PASSAGES]

# RANKING CRITERIA

Rank each passage on a scale of 1-5 (1=lowest, 5=highest) for each of the following criteria:

1.  **Direct Quote Support:** Does the passage contain a specific quote that directly supports the claim?
2.  **Claim Alignment:** How precisely does the evidence align with the core assertion of the claim?
3.  **Source Verifiability:** Is the source of the passage clearly identified and likely to be authoritative?

# CONSTRAINTS

- Do not rank a passage highly if it only tangentially relates to the claim.
- Penalize passages that are vague, lack specific details, or contain contradictory information.
- If a passage is entirely irrelevant, assign it a score of 0 for all criteria.
- Prioritize passages that provide direct, quotable evidence over those requiring inference.

# OUTPUT SCHEMA

Return a JSON object with a single key "ranked_evidence" containing an array of objects. Each object must have the following fields:

- `passage_id`: (string) The unique identifier of the passage from the input.
- `direct_quote_support_score`: (integer) 1-5
- `claim_alignment_score`: (integer) 1-5
- `source_verifiability_score`: (integer) 1-5
- `composite_score`: (number) The average of the three scores, rounded to one decimal place.
- `citation_worthiness`: (string) One of "high", "medium", "low", or "none".
- `rationale`: (string) A brief, specific explanation for the scores, referencing the passage content.

# EXAMPLE OUTPUT

```json
{
  "ranked_evidence": [
    {
      "passage_id": "doc_3_para_2",
      "direct_quote_support_score": 5,
      "claim_alignment_score": 5,
      "source_verifiability_score": 4,
      "composite_score": 4.7,
      "citation_worthiness": "high",
      "rationale": "Contains the direct quote '...the revenue increased by 20%...' which perfectly aligns with the claim. Source is a verified financial report."
    }
  ]
}

RISK LEVEL

[RISK_LEVEL]

To adapt this template, you must replace the placeholders. [USER_QUESTION] and [CLAIM_TO_SUPPORT] provide the necessary context for the model. [EVIDENCE_PASSAGES] should be a structured list of your retrieved documents, each with a unique ID and its text content. The [RISK_LEVEL] placeholder is a critical control; for high-stakes domains like healthcare or finance, you should append a final instruction such as: 'If the highest composite score is below 3.5, you must output an empty ranked_evidence array and state that no sufficiently strong evidence was found.' This prevents the model from fabricating a best-fit ranking from weak sources.

Before integrating this prompt into your application, test it with a golden dataset of claims and evidence where the correct ranking is known. Pay close attention to the citation_worthiness field in the output; this is your primary gating mechanism. In your application harness, you should parse the JSON output and only pass passages with a citation_worthiness of "high" or "medium" to the final answer generation step. Any other result should trigger a fallback, such as a request for more context or a refusal to answer.

IMPLEMENTATION TABLE

Prompt Variables

Required inputs for the Evidence Ranking for Citation Selection prompt. Each placeholder must be populated before the prompt is sent. Validation notes describe how to check that the input is well-formed and safe before execution.

PlaceholderPurposeExampleValidation Notes

[QUERY]

The user question or claim that evidence must support

What were the key drivers of Q3 revenue growth?

Must be non-empty string under 2000 chars. Reject if contains only whitespace or is a generic greeting.

[RETRIEVED_PASSAGES]

Unranked set of candidate evidence passages from retrieval

See schema: array of {passage_id, text, source_title, pub_date, author}

Must be valid JSON array with 2-100 objects. Each object requires passage_id and text fields. Reject if empty array.

[CITATION_REQUIREMENTS]

Rules for what makes a passage citation-worthy

Require direct quote support, verifiable source, and claim alignment

Must be a non-empty string or structured object defining min criteria. Default to 'direct quote support required' if null.

[OUTPUT_SCHEMA]

Expected JSON structure for ranked citation-ready passages

See schema: {ranked_passages: [{passage_id, rank, strength_tier, citation_span, rationale}]}

Must be a valid JSON Schema object or reference. Validate parseable before prompt assembly. Reject if schema allows unconstrained output.

[MAX_CITATIONS]

Upper bound on number of passages to select for citation

5

Must be integer between 1 and 20. Default to 5 if null. Reject if negative or zero.

[SOURCE_TRUST_TIERS]

Optional mapping of source domains or authors to authority levels

{'peer-reviewed': 'high', 'official-gov': 'high', 'blog': 'low'}

If provided, must be valid JSON object. Null allowed. If null, prompt uses content-based authority assessment only.

[CONFIDENCE_THRESHOLD]

Minimum confidence score for a passage to be included in citation set

0.7

Must be float between 0.0 and 1.0. Default to 0.5 if null. Passages below threshold are excluded from output.

PRACTICAL GUARDRAILS

Common Failure Modes

Evidence ranking pipelines break in predictable ways. These cards cover the most frequent failure modes in citation selection and how to guard against them before they reach users.

01

Position Bias Skews Top Ranks

What to watch: Models overweight passages that appear first in the context window, assigning higher relevance scores to early items regardless of actual quality. This produces rankings that reflect input order rather than evidence strength. Guardrail: Randomize passage order before ranking, run multiple ranking passes with shuffled inputs, and compare score stability across permutations. Flag rankings where position correlates with score above a threshold.

02

Length Bias Favors Verbose Passages

What to watch: Longer passages receive inflated relevance scores because they contain more tokens that match query terms, not because they contain stronger evidence. Short, precise passages get buried. Guardrail: Normalize scores by passage length or apply a brevity bonus. Test ranking outputs with length-controlled passage sets where the shortest passage contains the strongest evidence. If the short passage doesn't rank first, length bias is active.

03

Authority Overfitting Ignores Content

What to watch: The model overweights source authority metadata and ignores whether the passage actually supports the claim. High-authority sources with tangential content outrank lower-authority sources with direct evidence. Guardrail: Separate authority scoring from content relevance scoring. Run a two-pass ranking: first by content-to-claim alignment, then apply authority as a tiebreaker or secondary weight. Test with high-authority irrelevant passages to confirm they don't displace relevant evidence.

04

Query Misalignment Produces Thematic Drift

What to watch: Passages that share topic keywords but address a different question or angle receive high scores. The ranking looks thematically relevant but fails to answer the specific query. Common with broad retrieval sets. Guardrail: Include the exact question or claim in the ranking prompt, not just keywords. Add a specificity check: require the model to identify which sentence in each passage directly addresses the query. Discard passages where no specific sentence maps to the question.

05

Redundancy Collapse Narrows Evidence Diversity

What to watch: Top-K selection returns multiple passages from the same source or near-duplicate content, crowding out diverse evidence. The ranking looks strong but the answer generation sees only one perspective repeated. Guardrail: Add a diversity constraint to the ranking prompt. After selecting each passage, require the model to check whether the next candidate adds new information or repeats existing selections. Implement source-level deduplication before final ranking.

06

Confidence Inflation Masks Weak Evidence

What to watch: All passages receive medium-to-high confidence scores even when evidence is thin, tangential, or speculative. The ranking pipeline never signals insufficiency, so downstream answer generation proceeds with weak grounding. Guardrail: Calibrate confidence tiers with explicit anchors. Define what "low confidence" looks like with examples. Add a minimum confidence threshold below which passages are excluded. Test with retrieval sets containing only weak evidence and verify the pipeline flags insufficiency rather than ranking garbage.

IMPLEMENTATION TABLE

Evaluation Rubric

Criteria for testing the quality and reliability of the Evidence Ranking for Citation Selection Prompt before production deployment. Each criterion includes a concrete pass standard, a clear failure signal, and a test method that can be automated in an eval harness.

CriterionPass StandardFailure SignalTest Method

Citation Span Accuracy

Every selected passage includes a direct quote span that matches the source text exactly

Selected quote contains hallucinated words, truncated sentences that change meaning, or spans not found in the source

String-inclusion check: verify each [CITATION_SPAN] substring exists in the corresponding [SOURCE_TEXT]

Claim-to-Evidence Alignment

Each ranked passage directly supports the specific [CLAIM] provided, with alignment score >= 0.8 on a 0-1 scale

Passage ranked highly but addresses a related topic rather than the specific claim, or alignment rationale is circular

LLM-as-judge pairwise comparison: present [CLAIM] and two passages, verify the higher-ranked passage is more directly supportive

Ranking Stability Under Shuffling

Top-3 passages remain identical regardless of input passage order, with rank correlation > 0.95 across 5 shuffled runs

Top-K selection changes significantly when passage order is randomized, indicating position bias

Shuffle test: run ranking 5 times with randomized [PASSAGE_LIST] order, compute Kendall tau correlation between runs

Source Verifiability Flagging

Passages from unverifiable or dead-link sources are flagged with verifiability=false and ranked below verifiable sources

Unverifiable source ranked above a verifiable source with equivalent relevance, or verifiability field missing from output

Schema check: confirm [VERIFIABILITY] field present and boolean; inject known-dead-link passage and verify it is not in top-3

Output Schema Compliance

Output matches the defined [OUTPUT_SCHEMA] exactly: all required fields present, correct types, no extra fields

Missing [RANK], [PASSAGE_ID], or [RATIONALE] fields; rank values not integers; confidence outside 0-1 range

JSON Schema validator: validate output against the contract schema; fail on any validation error

Confidence Score Calibration

Confidence scores correlate with actual citation usefulness: high-confidence passages (>0.8) are cited in final answer >= 90% of the time

High confidence assigned to passages that downstream answer generation ignores or that contain contradictory information

Holdout test: compare [CONFIDENCE] scores against downstream citation rate in a golden dataset of 50+ examples

Empty Retrieval Handling

When [PASSAGE_LIST] is empty or contains only irrelevant passages, output is an empty ranked list with a gap explanation, not a hallucinated ranking

Model invents plausible-sounding passages, ranks irrelevant content as high-confidence, or returns a non-empty list

Boundary test: submit empty [PASSAGE_LIST] and list of off-topic passages; verify output is empty array with [GAP_EXPLANATION] populated

Rationale Grounding

Every [RATIONALE] field references specific content from the passage (quoted text, entity, or data point), not generic statements

Rationale says 'this passage is relevant' without citing what specifically supports the claim; rationale repeats the claim verbatim

Regex check: verify each [RATIONALE] contains at least one quoted substring from the corresponding [PASSAGE_TEXT] or a specific entity reference

ADAPTATION OPTIONS

Adapt This Prompt

How to adapt

Use the base prompt with a simple list output instead of a strict JSON schema. Focus on getting ranking logic right before adding validation. Remove the [OUTPUT_SCHEMA] block and ask for a numbered list with brief rationale per passage.

Prompt snippet

code
Rank the following evidence passages by how well they support the claim [CLAIM].
Return a numbered list from strongest to weakest. For each passage, include:
- Passage ID
- Strength tier (Direct Quote Support / Partial Support / Weak / No Support)
- One sentence explaining your ranking

Passages:
[PASSAGES]

Watch for

  • Position bias where earlier passages get higher ranks regardless of content
  • Overly generous "Partial Support" assignments for vague passages
  • Missing passage IDs when the model reorders but drops identifiers
Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.