Evidence Selection for Comparative Questions Prompt

This prompt is designed for RAG pipelines that must answer comparative questions such as 'Compare Product A and Product B on security features' or 'What are the differences between Framework X and Framework Y for real-time data processing?' Standard evidence selection prompts often exhibit a proximity bias, selecting passages that heavily favor one entity while ignoring another. This prompt enforces balanced representation by requiring the model to select and rank evidence for each entity in the comparison before synthesis.

Use this prompt when your application must produce fair, multi-sided comparisons from a knowledge base, when you need to detect missing evidence for one side of a comparison, and when downstream answer generation depends on unbiased evidence selection. It is particularly valuable for product comparison features, competitive analysis tools, framework evaluation assistants, and any Q&A system where users expect both sides of a comparison to be represented. The prompt works best with retrieval sets that contain passages about all entities being compared; if your knowledge base is heavily skewed toward one entity, the prompt will surface that gap rather than fabricate balance.

Do not use this prompt for single-entity questions ('What are the features of Product A?'), for ranking tasks that don't involve comparison, or when your context window is too small to hold evidence for multiple entities. It is also not a substitute for retrieval quality—if your retriever cannot find relevant passages for one side of the comparison, the prompt will correctly flag the gap but cannot invent missing evidence. For high-stakes comparisons in regulated domains, always pair this prompt with human review of the balanced evidence set before answer synthesis.

This prompt template performs balanced evidence selection for comparative questions. It is designed to be inserted into your RAG pipeline after retrieval and before answer synthesis. The prompt forces the model to explicitly select and rank passages that fairly represent all entities mentioned in the query, not just the first or most prominent one. Use it when your product surfaces side-by-side comparisons—product comparisons, vendor evaluations, protocol differences, or any multi-entity Q&A where selection bias toward a single subject would break the user experience.

textCopy
You are an evidence selection system for a comparative question-answering application. Your job is to select and rank the most relevant passages from the retrieved context to support a fair, balanced comparison.

## INPUT
Query: [QUERY]
Entities mentioned in the query: [ENTITIES]
Retrieved passages: [PASSAGES]

## CONSTRAINTS
- You must select at least one passage for each entity in [ENTITIES] if any relevant passage exists.
- If no relevant passage exists for an entity, mark it as MISSING and explain why.
- Do not select more than [MAX_PASSAGES] total passages.
- Prefer passages that directly compare entities over passages that discuss only one.
- When multiple passages cover the same fact, select the most credible source based on [CREDIBILITY_CRITERIA].
- Exclude passages that are duplicates, near-duplicates, or contain no comparative information.

## OUTPUT_SCHEMA
Return a JSON object with this structure:
{
  "selected_passages": [
    {
      "passage_id": "string",
      "entity_coverage": ["entity_name"],
      "relevance_score": 0.0-1.0,
      "selection_reason": "string",
      "passage_text": "string"
    }
  ],
  "entity_balance": {
    "entity_name": {
      "passage_count": integer,
      "status": "COVERED" | "UNDERREPRESENTED" | "MISSING"
    }
  },
  "excluded_passages": [
    {
      "passage_id": "string",
      "exclusion_reason": "string"
    }
  ],
  "selection_summary": "string"
}

## RISK_LEVEL
[RISK_LEVEL]

Adaptation notes: Replace [CREDIBILITY_CRITERIA] with your domain-specific rules—for example, prefer peer-reviewed sources, official documentation, or recency-weighted authority. The [ENTITIES] field should be populated by an upstream entity extraction step; if you don't have one, add an instruction for the model to extract entities from the query first. The [RISK_LEVEL] field controls how conservative the selection should be: set it to HIGH for regulated domains where missing evidence must block answer generation, or LOW for exploratory comparisons where partial coverage is acceptable. After selection, validate the output JSON against the schema before passing it to your synthesis prompt. If the entity_balance field shows any entity as MISSING and your risk level is HIGH, route to a human reviewer or trigger an abstention response rather than synthesizing an unbalanced answer.

This prompt is designed to sit between retrieval and answer synthesis in a comparative Q&A pipeline. It expects a list of entities to compare (e.g., 'Product A, Product B, Product C') and a set of retrieved passages. The harness must enforce representation balance: if the prompt returns evidence skewed toward one entity, the harness should either re-query retrieval with entity-specific filters or flag the output for human review before synthesis proceeds. Do not pass unbalanced evidence selection directly to an answer generation prompt without inspection.

Integration follows a standard RAG pattern: the application receives a comparative query, extracts entity names via a lightweight NER or structured extraction call, retrieves passages for each entity (ideally in parallel with equal top_k limits), concatenates them into the [RETRIEVED_PASSAGES] placeholder, and invokes this prompt. The output schema should be a JSON array of selected passages with entity, passage_id, relevance_score, and selection_rationale fields. Validate the response with a JSON schema validator. If the output fails schema validation, retry once with the error message appended to the prompt. If it fails twice, log the raw output and escalate to a human reviewer.

The most common production failure mode is entity imbalance: the model selects 8 passages for one product and 1 for another. Implement a post-processing balance check that counts selections per entity and rejects outputs where any entity has fewer than min_passages_per_entity (typically 2). When imbalance is detected, do not silently proceed. Instead, route to a fallback path: either re-run the prompt with explicit per-entity passage quotas in the [CONSTRAINTS] block, or use a simpler per-entity selection prompt for each entity individually and merge results. Log imbalance events with entity counts and query text for prompt improvement cycles.

For model choice, use a model with strong instruction-following and JSON mode support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature=0 or very low to maximize selection consistency. If your retrieval system returns more than 20 passages, pre-filter to the top 15 by vector similarity score before invoking this prompt to stay within context window limits and reduce selection noise. For high-stakes domains (legal, clinical, financial), always require human review of the selected evidence set before answer synthesis, and log the full selection rationale for audit trails.