Source Trustworthiness Batch Scoring Prompt for Pipelines

This playbook is for engineering teams running evidence ingestion pipelines that process large document collections. The prompt scores multiple sources in a single pass, producing structured output optimized for database insertion. Use it when you need to assign credibility scores, authority tiers, and confidence calibrations to hundreds or thousands of sources before they enter your RAG retrieval index. This prompt assumes you have already extracted source metadata (URL, domain, author, publication date, publisher) and need batch-level consistency checks across the entire set.

The ideal user is a pipeline engineer or trust-and-safety developer who controls the ingestion path and needs deterministic, comparable scores across sources. You should have a metadata store or document database ready to receive structured JSON output. The prompt works best when sources share a common domain context—scoring a mix of academic papers, news articles, and government reports in one batch is fine, but the scoring criteria must apply uniformly. If your sources span radically different authority models (e.g., legal precedents vs. social media posts), split them into separate batches with domain-specific constraints.

Do not use this prompt for real-time single-source scoring during query time. The batch design trades latency for throughput and cross-source consistency checks. For latency-sensitive paths where a user submits a query and you need to score one retrieved source immediately, use the Multi-Factor Source Credibility Scoring Prompt Template instead. Also avoid this prompt when source metadata is incomplete—if you lack publication dates, author information, or publisher domains for most sources, the batch consistency checks will produce low-confidence scores across the board, wasting compute and storage.

Before running this prompt at scale, validate your metadata extraction pipeline. Missing or malformed fields will degrade scoring quality. Start with a small batch of 20-50 sources, review the output distribution, and calibrate thresholds before processing thousands of records. The structured output includes confidence intervals and flag fields specifically designed for downstream filtering, so wire those into your database schema and retrieval weighting logic from the start.

This prompt template is designed for evidence ingestion pipelines that process large document collections. It scores multiple sources in a single model call, producing structured JSON output optimized for direct insertion into your trustworthiness database. The prompt enforces batch-level consistency checks and confidence calibration, ensuring that scores across sources remain comparable rather than drifting with each independent evaluation.

textCopy
SYSTEM:
You are a source trustworthiness batch scorer for an evidence ingestion pipeline. Your job is to evaluate multiple sources in a single pass and return structured scores suitable for database insertion. Apply consistent standards across all sources in the batch. Do not let the order of sources influence your scoring.

SCORING DIMENSIONS:
For each source, evaluate these dimensions on a 0.0 to 1.0 scale:
- authority: Institutional or author expertise signals (credentials, affiliation, citation patterns)
- accuracy: Factual precision and absence of verifiable errors
- objectivity: Freedom from commercial, political, or ideological bias
- recency: Currency relative to [TEMPORAL_BASELINE_DATE]
- completeness: Coverage depth and absence of material omissions

CONFIDENCE CALIBRATION:
- Assign a confidence_score (0.0 to 1.0) reflecting how much evidence you had to evaluate the source
- If metadata is sparse, reduce confidence rather than guessing
- Flag sources with confidence below [CONFIDENCE_THRESHOLD] for human review

INPUT FORMAT:
You will receive a JSON array of source objects. Each object contains:
- source_id: Unique identifier
- url: Source URL (may be empty)
- domain: Publisher domain
- title: Document or article title
- author: Author name(s) (may be empty)
- publication_date: ISO date string (may be empty)
- publisher: Organization name (may be empty)
- content_snippet: First 500 words of the source text
- metadata: Arbitrary key-value pairs from the retrieval system

[INPUT_BATCH]

OUTPUT FORMAT:
Return a JSON object with this exact structure:
{
  "batch_id": "[BATCH_ID]",
  "scored_at": "ISO timestamp",
  "sources": [
    {
      "source_id": "string",
      "authority": 0.0,
      "accuracy": 0.0,
      "objectivity": 0.0,
      "recency": 0.0,
      "completeness": 0.0,
      "composite_score": 0.0,
      "confidence_score": 0.0,
      "flags": ["string"],
      "explanation": "Brief justification citing specific signals"
    }
  ],
  "batch_consistency_notes": ["string"]
}

RULES:
1. Compute composite_score as the weighted average: authority*0.30 + accuracy*0.30 + objectivity*0.15 + recency*0.15 + completeness*0.10
2. Round all scores to 2 decimal places
3. Use flags array for: "low_confidence", "missing_date", "anonymous_author", "commercial_bias_detected", "political_bias_detected", "retracted_or_corrected", "ai_generated_suspected", "single_source_claim", "predatory_publisher_suspected"
4. In batch_consistency_notes, document any calibration adjustments you made to keep scores comparable across sources
5. If a source lacks sufficient information to score a dimension, use null for that dimension and reduce confidence_score accordingly
6. Do not invent information. If author is unknown, do not guess
7. Return ONLY valid JSON. No markdown fences, no commentary

Adaptation guidance: Replace [TEMPORAL_BASELINE_DATE] with your pipeline's reference date for recency scoring. Set [CONFIDENCE_THRESHOLD] to your organization's risk tolerance—0.4 is a common starting point. The [INPUT_BATCH] placeholder should be replaced by your pipeline's JSON array of source objects. The dimension weights in the composite score formula can be adjusted per domain; for legal workflows, increase authority weight; for news, increase recency weight. The flags enum should be extended to match your specific trust-and-safety taxonomy.

Validation before production: Before deploying this prompt, run it against a golden dataset of 20–30 sources with known trustworthiness labels. Verify that composite scores correlate with your ground-truth labels. Check that batch_consistency_notes appear when score distributions drift. If the model overweights a single dimension, adjust the weights or add explicit calibration instructions. For high-stakes pipelines, route sources flagged with low_confidence or retracted_or_corrected to a human review queue before they enter your evidence store.

The batch scoring prompt is designed to run inside a pipeline stage that processes multiple source records at once, not as a single-source interactive call. The typical integration point is after document retrieval and metadata extraction but before evidence ranking. A pipeline worker receives a batch of source objects, each containing a source_id, url, title, publisher, publication_date, author_metadata, and any available content_snippet. The worker assembles the prompt by injecting these records into the [SOURCE_BATCH] placeholder as a JSON array, sets [BATCH_SIZE] to the number of records (typically 5–20 to balance throughput with scoring quality), and includes the [OUTPUT_SCHEMA] that defines the expected JSON structure with fields like trust_score, authority_tier, recency_flag, bias_indicators, and confidence_calibration.

Validation must happen in two layers. First, parse the model response and confirm every source_id from the input batch appears in the output with a valid trust_score between 0 and 1. Missing records or out-of-range scores should trigger a structured retry: resend only the failed records with an error context message like 'Previous response missing scores for source_ids: [X, Y]. Return complete batch.' Second, run a consistency check across the batch. If the model assigns authority_tier: 'high' to a source with an unrecognized domain and no institutional affiliation signals, flag that record for human review rather than inserting it directly. For high-risk domains such as healthcare or legal evidence pipelines, require human approval on any record where confidence_calibration falls below 0.7 or where bias_indicators include 'undisclosed_funding' or 'retraction_possible'. Log every batch run with the model version, timestamp, batch size, validation pass/fail counts, and any records escalated for review.

Model choice matters for this workload. Use a model with strong structured output support and low variance across repeated runs—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models that may drift on the scoring rubric across batches. Set temperature to 0 or near-zero to maximize consistency. If your pipeline processes thousands of sources daily, implement a caching layer: sources with unchanged metadata and a recent trust score (within 30 days) can skip rescoring unless a force_refresh flag is set. For cost control, batch as many records as fit within the model's context window without truncation, but never exceed 30 records per batch—larger batches degrade per-source attention and increase the risk of skipped records. Wire the validated output into your evidence database with columns for each score field, plus scored_at, model_version, and review_status. This schema lets downstream ranking queries filter by authority_tier and trust_score thresholds without rerunning the prompt.