Source Credibility Scoring Prompt Template

This prompt is for verification engineers building automated fact-checking pipelines who need to score a source's credibility before using it as evidence. It produces a structured credibility score with domain-specific authority, recency, and bias dimensions. Use it when your pipeline must dynamically weight evidence sources and explain why one source was preferred over another. The prompt assumes you have already retrieved or identified a candidate source and have its metadata available—it does not perform retrieval, claim extraction, or evidence matching. For those tasks, see the sibling playbooks on claim extraction and evidence matching.

The ideal user is an engineer or pipeline operator who needs a repeatable, auditable credibility signal that can feed downstream decisions: evidence ranking, human review routing, or automated verification thresholds. The prompt expects structured inputs including source title, URL, publication date, author, publisher, and the claim domain being verified. Without this context, the model will produce unreliable scores based on internal knowledge cutoffs rather than the specific source and claim at hand. Always provide the claim domain (e.g., 'clinical medicine,' 'macroeconomic policy,' 'semiconductor fabrication') because authority is domain-relative—a Nobel laureate in physics is not an authoritative source for immigration law.

Do not use this prompt when you lack source metadata, when the source is a first-person account requiring witness credibility assessment rather than institutional authority scoring, or when the claim domain is so niche that no recognizable authority hierarchy exists. In those cases, fall back to the Evidence Directness Assessment Prompt or route directly to human review with a structured handoff. Also avoid this prompt for real-time breaking news verification where source credibility is still forming—the recency dimension will over-penalize emerging sources that may later prove authoritative. For time-sensitive claims, pair this prompt with the Recency Evaluation Prompt and set explicit staleness thresholds per domain.

This section provides a copy-ready prompt template that you can drop into your verification pipeline. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to wire into an application layer that populates source metadata, claim context, and output format requirements before each model call. The prompt is designed to produce a structured credibility score with explicit reasoning across multiple dimensions, not a single opaque number.

textCopy
You are a source credibility assessor for an automated fact-checking pipeline. Your task is to evaluate the credibility of a given source for a specific claim domain and produce a structured assessment.

## INPUT
Source Metadata:
[SOURCE_METADATA]

Claim Domain:
[CLAIM_DOMAIN]

Claim Context (optional):
[CLAIM_CONTEXT]

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "source_identifier": "string",
  "overall_credibility_score": number (0.0 to 1.0),
  "dimension_scores": {
    "domain_authority": {
      "score": number (0.0 to 1.0),
      "rationale": "string explaining why this source is or is not authoritative for the claim domain",
      "evidence": ["specific signals from source metadata that support this score"]
    },
    "recency": {
      "score": number (0.0 to 1.0),
      "rationale": "string explaining temporal relevance for the claim domain",
      "publication_date": "ISO 8601 date or null if unknown",
      "staleness_threshold_exceeded": boolean
    },
    "methodology_transparency": {
      "score": number (0.0 to 1.0),
      "rationale": "string explaining whether methods, sample sizes, and limitations are disclosed",
      "gaps_flagged": ["specific missing disclosures"]
    },
    "independence": {
      "score": number (0.0 to 1.0),
      "rationale": "string explaining source independence from the claim subject",
      "conflict_of_interest_detected": boolean,
      "funding_disclosures": ["any funding sources identified"]
    },
    "corroboration_potential": {
      "score": number (0.0 to 1.0),
      "rationale": "string explaining whether this source type can be independently verified",
      "source_type": "primary | secondary | tertiary | unknown"
    }
  },
  "limitations": ["known limitations or caveats about this assessment"],
  "recommended_usage": "string describing how this source should be weighted in verification",
  "confidence_in_assessment": number (0.0 to 1.0)
}

## CONSTRAINTS
- If source metadata is sparse, set confidence_in_assessment low and flag missing fields in limitations.
- Do not invent metadata. If a field is unknown, use null or an empty array.
- For recency, apply domain-appropriate staleness thresholds: [RECENCY_THRESHOLDS]
- For domain authority, prioritize these source types: [AUTHORITY_PRIORITIES]
- If the source is a press release, corporate blog, or sponsored content, flag it in independence and reduce the score accordingly.
- If the source is anonymous or unattributed, set corroboration_potential low and note the constraint.
- Do not produce scores without explicit rationale tied to provided metadata.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Return only the JSON object. Do not include markdown fences, explanations, or commentary outside the JSON.

To adapt this template, replace each bracketed placeholder with live data from your pipeline. [SOURCE_METADATA] should include all available fields: publication name, author credentials, date, URL, funding disclosures, methodology notes, and source type. [CLAIM_DOMAIN] is a short label like "clinical trial results" or "economic indicators" that drives domain-specific authority and recency logic. [RECENCY_THRESHOLDS] and [AUTHORITY_PRIORITIES] let you inject domain rules without rewriting the prompt. [FEW_SHOT_EXAMPLES] should include 2-3 scored examples that calibrate the model to your expected score distribution. [RISK_LEVEL] controls the strictness of the assessment: use "high" for regulated domains requiring conservative scoring, "medium" for general verification, and "low" for exploratory research where false positives are acceptable.

Before deploying this prompt, validate that your application layer can parse the JSON output and handle missing or null fields gracefully. Run the prompt against your eval suite using the companion Source Credibility Eval Check prompt to detect score drift, overconfidence on sparse metadata, and domain mismatch failures. If the prompt will be used in high-stakes verification, route outputs with confidence_in_assessment below 0.7 to human review. Do not treat the overall_credibility_score as a precise measurement; it is a structured signal for downstream weighting, not a ground-truth label.

The Source Credibility Scoring prompt is designed to be called as a single step within a larger fact-checking or evidence-weighting pipeline. It expects structured source metadata as input and returns a structured credibility score. In production, you should not call this prompt in isolation; wrap it in a harness that validates the output schema, enforces score thresholds, and routes low-confidence or edge-case results for human review. The prompt itself is stateless, so all necessary context—source URL, publication date, author credentials, domain of the claim being verified—must be assembled by the application layer before the model call.

Validation and retry logic: The output must conform to a strict JSON schema with required fields for credibility_score (0.0–1.0), dimension_scores (authority, recency, bias, transparency), and evidence_chain. Implement a post-generation validator that checks field presence, type correctness, and range constraints. If validation fails, retry once with the same prompt plus the validation error message appended as a [REPAIR_INSTRUCTION]. If the second attempt also fails, log the failure, fall back to a default low-confidence score, and flag the source for manual review. Do not retry more than twice—diminishing returns and latency costs escalate quickly.

Model choice and latency budgeting: For high-throughput pipelines processing hundreds of sources per claim, use a fast, cost-efficient model (e.g., GPT-4o-mini, Claude Haiku) with a strict token budget. Reserve larger models (GPT-4o, Claude Sonnet) for sources flagged as high-stakes or where the initial score falls in an ambiguity band (e.g., 0.4–0.6). Implement a model router that checks the claim's risk level and the source's metadata completeness before selecting the model tier. Log every model call with the prompt version, model ID, latency, and token usage for cost attribution and debugging.

Human review gating: Any source that receives a credibility score below a configurable threshold (start with 0.5) or has missing critical metadata (no publication date, no author, no domain information) should be routed to a review queue. The harness should package the raw source metadata, the model's full JSON output, and the claim context into a review packet. Human reviewers can override the score, and those overrides should be logged as training data for future fine-tuning or prompt calibration. Never auto-accept a low-credibility source without human confirmation in high-stakes domains like healthcare, legal, or financial verification.

Observability and drift detection: Instrument the harness to emit structured logs for every prompt call, including the input metadata hash, output scores, validation pass/fail status, retry count, and review routing decision. Monitor score distributions over time—sudden shifts in average credibility scores or dimension score patterns often indicate prompt drift, model behavior changes, or upstream metadata quality degradation. Set up alerts for validation failure rate spikes and review queue backlog growth. Store prompt versions alongside outputs so you can correlate score changes with prompt updates during incident analysis.