Source Credibility Weighting Prompt for Conflicts

This prompt is for RAG system builders who need to resolve factual conflicts by evaluating the credibility of each source, not by ignoring the disagreement or picking the first result. The job-to-be-done is: given a set of conflicting claims extracted from multiple retrieved documents, assign a credibility weight to each source based on its metadata—recency, authoritativeness, domain relevance, and publication type—then rank the conflicting claims by weighted evidence strength. The ideal user is an AI engineer or knowledge base operator who already has source metadata available (publication dates, author roles, domain tags, document types) and needs a repeatable, auditable method for surfacing which claim should carry more weight in the final answer. This is not a fact-checking prompt; it assumes the claims are already extracted and that the conflict has been detected upstream by a conflict detection or contradiction flagging prompt.

Use this prompt when your retrieval pipeline returns documents of varying quality and you need to programmatically decide which sources to trust more. It works best when you can supply structured metadata per source: source_id, publication_date, author_role (e.g., 'regulator', 'vendor', 'peer-reviewed researcher', 'internal team'), domain_match (whether the source's domain aligns with the question's domain), and document_type (e.g., 'policy', 'report', 'specification', 'blog'). The prompt outputs a ranked list of claims with per-source credibility scores and a brief weighting rationale, making the decision traceable for downstream answer generation or human review. Wire this into a conflict resolution step that runs after contradiction detection and before answer synthesis, so the final RAG answer can weight evidence appropriately rather than treating all sources as equal.

Do not use this prompt when you lack source metadata—credibility weighting without metadata is guesswork dressed as analysis. Do not use it as a standalone fact-checker; it does not verify claims against ground truth, only weights them by source attributes. Avoid this prompt when the conflict is temporal and a simple recency rule (newer supersedes older) suffices—use the Temporal Conflict Resolution Prompt instead. If the conflict stems from definitional differences or methodology variance rather than source credibility, use the Source Conflict Root Cause Analysis Prompt. For high-stakes domains like healthcare or legal, always route the weighted output to human review before it influences a final answer; credibility scores are decision support, not automated adjudication. The next step after reading this section is to inventory your available source metadata and confirm you can populate the required input fields before adapting the prompt template.

The prompt below is designed to be dropped into a RAG pipeline after retrieval but before final answer synthesis. It takes a set of conflicting claims and their source metadata, then produces a weighted ranking with explicit credibility rationale. Use it when your system must resolve contradictions programmatically rather than surfacing all conflicts to the user or abstaining entirely. The template expects source metadata to be available—if your retrieval pipeline doesn't capture publication dates, author credentials, or domain authority signals, you'll need to add that extraction step first.

textCopy
You are a source credibility analyst for a RAG system. Your task is to evaluate conflicting claims from multiple sources and assign credibility weights based on source authority, recency, and domain relevance.

## INPUT

Claims and sources with conflicts:
[CONFLICTING_CLAIMS]

Source metadata for each source referenced above:
[SOURCE_METADATA]

## CREDIBILITY CRITERIA

Evaluate each source on these dimensions, scoring 1-5 for each:

1. **Authoritativeness**: Is the source a recognized authority in this domain? Consider publisher reputation, author credentials, institutional affiliation, and peer review status.
2. **Recency**: How current is the source relative to the claims being evaluated? Newer sources on fast-changing topics score higher; archival sources on historical facts remain credible.
3. **Domain Relevance**: How directly does the source's expertise match the specific claim domain? A medical journal scores higher on health claims than a general news outlet.
4. **Evidence Quality**: Does the source provide primary evidence, citations, methodology, or data? Sources with traceable evidence score higher than those making unsupported assertions.
5. **Consensus Alignment**: How well does the source align with the broader consensus across all provided sources? Outliers require stronger authority to maintain credibility.

## OUTPUT FORMAT

Return a JSON object with this exact schema:

{
  "source_credibility_scores": [
    {
      "source_id": "string",
      "source_name": "string",
      "authoritativeness": 1-5,
      "recency": 1-5,
      "domain_relevance": 1-5,
      "evidence_quality": 1-5,
      "consensus_alignment": 1-5,
      "composite_credibility": 0.0-1.0,
      "credibility_rationale": "string explaining the scores"
    }
  ],
  "claim_resolution": [
    {
      "claim": "string",
      "conflicting_sources": ["source_id"],
      "weighted_resolution": "string stating the most credible position",
      "confidence": 0.0-1.0,
      "weighting_rationale": "string explaining how credibility weights determined the resolution"
    }
  ],
  "unresolvable_conflicts": [
    {
      "claim": "string",
      "reason": "string explaining why credibility weighting could not resolve this conflict"
    }
  ]
}

## CONSTRAINTS

[CONSTRAINTS]

## RISK LEVEL

[RISK_LEVEL]

Adapt this template by adjusting the credibility criteria to match your domain's authority signals. For legal use cases, add criteria for court hierarchy and jurisdiction relevance. For financial applications, weight regulatory filings and audited statements more heavily. For scientific domains, prioritize peer-reviewed sources and sample sizes. The composite credibility score formula should be tuned to your risk tolerance—a simple average works for low-stakes applications, but high-stakes domains may need weighted formulas where authoritativeness dominates. Always run this prompt through your eval suite with known-conflict test cases before deployment, and implement a validator that rejects outputs where confidence falls below your minimum threshold or where unresolvable_conflicts contains claims your system must answer.

This prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It receives a set of conflicting claims and their source metadata, then outputs a weighted ranking with credibility scores. The harness must supply structured source metadata—publication date, author authority level, domain relevance tags, and citation count—as part of the [SOURCES] input. Without this metadata, the model cannot perform meaningful weighting and will default to surface-level heuristics. The prompt expects a JSON array of source objects, each containing the conflicting claim, source identifier, and the metadata fields defined in the [SOURCE_SCHEMA] placeholder.

Wire this prompt as a synchronous processing step with a strict output contract. After the model returns weighted results, validate the JSON structure against the expected schema: each source must have a credibility_score (0.0–1.0), a weight (0.0–1.0), and a rationale string. Reject outputs where scores fall outside bounds, where rationales are empty, or where the ranking contradicts explicit precedence rules (e.g., a peer-reviewed source from last month outranked by a blog post from five years ago without justification). On validation failure, retry once with the error message appended to the [CONSTRAINTS] field. If the second attempt also fails, log the failure and route to a human review queue rather than silently proceeding with unweighted evidence.

For high-stakes domains—legal, clinical, financial—add a confidence threshold gate. If the model's self-reported confidence in its weighting (requested via a weighting_confidence field in the output schema) falls below 0.7, or if the top two sources have credibility scores within 0.1 of each other, escalate to human review. Use a model with strong instruction-following and JSON output reliability, such as gpt-4o or claude-3-5-sonnet, and set temperature=0 to minimize ranking instability. Log every weighting decision with the input sources, output scores, validator results, and escalation outcome. This audit trail is essential for debugging credibility drift and for compliance reviews where source weighting decisions must be explainable.

Do not use this prompt as a standalone conflict resolver. It weights sources but does not synthesize an answer. The downstream answer generation prompt must receive the weighted ranking and be instructed to prefer higher-weighted sources when claims conflict. Avoid the temptation to skip validation by trusting raw model output—credibility scores are subjective and models can overweight recency or underweight domain expertise without explicit calibration. Start with a golden dataset of 20–30 known source pairs with expected credibility rankings, run this prompt against them, and measure rank correlation (Spearman's ρ) before deploying. Recalibrate when source metadata schemas change or when new document types enter the retrieval corpus.