Domain Authority Assessment Prompt for Evidence Weighting

This prompt is designed for a specific job: dynamically weighting the credibility of an evidence source based on how well its domain of expertise matches the domain of a claim you need to verify. It is not a general-purpose credibility scorer. Use it inside a source authority pipeline, directly before evidence matching, to ensure that a medical journal is weighted more heavily for a clinical claim and a financial regulator's report is weighted more heavily for an audit claim. The ideal user is a verification system architect or pipeline engineer who already has structured source metadata (publication, author, type) and a clearly defined claim domain, and who needs a programmatic, explainable score to feed into downstream evidence ranking.

To use this prompt effectively, you must provide two key inputs: a structured description of the source and a clearly defined claim domain. The prompt then produces a domain-match score and a human-readable explanation. This output is a signal, not a final decision. It should be combined with other assessments like recency, bias, and directness before a source is accepted or rejected. Do not use this prompt for binary source acceptance decisions, for evaluating recency or factual accuracy, or for scoring a source's credibility in a vacuum without a specific claim domain. Using it without a claim domain will produce over-generalized and unreliable authority scores.

The primary failure mode of this prompt is domain over-generalization, where a source with broad prestige (e.g., a major newspaper) is scored as highly authoritative for a specialized scientific claim. The prompt's instructions and constraints are specifically designed to penalize this mismatch. A secondary failure mode is domain mismatch, where the prompt fails to recognize that a source's authority is irrelevant to the claim's domain. Always test this prompt with a golden dataset of source-claim pairs that includes these tricky cases. After running this assessment, the next step in your pipeline should be evidence matching, where the weighted source is used to evaluate the claim's veracity. If you need to assess a source's general credibility without a claim, use a general source credibility scoring prompt instead.

The following prompt template is designed to be pasted directly into your system instructions or as a user message in your verification pipeline. It instructs the model to act as a domain authority assessor, evaluating a given source against a specific claim topic. The core logic separates the source's general credibility from its specific expertise on the topic at hand, preventing a well-known generalist source from being incorrectly weighted as highly authoritative for a niche, specialized claim.

textCopy
You are a Domain Authority Assessment engine. Your task is to evaluate the authority of a single source in relation to a specific claim topic. You must produce a structured JSON output that includes a domain-match score and a clear, evidence-based explanation.

**INPUTS:**
- Source Metadata: [SOURCE_METADATA]
- Claim Topic: [CLAIM_TOPIC]
- Claim Domain: [CLAIM_DOMAIN]

**ASSESSMENT CRITERIA:**
1. **Institutional Authority:** Does the source have a formal mandate, recognized expertise, or established reputation in the [CLAIM_DOMAIN]? (e.g., a government agency for official statistics, a peer-reviewed journal for scientific findings).
2. **Author/Entity Expertise:** What are the specific credentials, track record, or affiliations of the author or publishing entity that relate directly to the [CLAIM_TOPIC]?
3. **Topic Relevance:** How directly does the source's core focus align with the specific [CLAIM_TOPIC]? Distinguish between a generalist source touching on the topic and a specialist source dedicated to it.
4. **Recency of Authority:** Is the source's authority current? An expert who hasn't published on the topic in 20 years may have diminished authority for a fast-moving field.

**OUTPUT_SCHEMA:**
You must return a single, valid JSON object with the following fields:
- `domain_authority_score` (integer, 1-5): A score where 1 is 'no domain authority' and 5 is 'definitive, top-tier domain authority'.
- `score_rationale` (string): A concise summary of the primary reason for the score.
- `authority_assessment` (object):
  - `institutional_authority` (string): Assessment based on Criterion 1.
  - `entity_expertise` (string): Assessment based on Criterion 2.
  - `topic_relevance` (string): Assessment based on Criterion 3.
  - `recency_note` (string or null): Assessment based on Criterion 4, or null if not applicable.
- `domain_mismatch_flags` (array of strings): List any detected mismatches where the source's general credibility could be mistaken for domain-specific authority. For example: ["High general credibility but no specific expertise in this domain", "Source is authoritative for a parent domain but not this specific sub-topic"].
- `comparative_authority_note` (string): A brief statement on how this source's authority would likely compare to a dedicated specialist source in [CLAIM_DOMAIN].

**CONSTRAINTS:**
- Do not confuse general brand recognition with domain-specific authority.
- If the source metadata is insufficient to make a confident assessment, set `domain_authority_score` to 1 and explain the missing information in the `score_rationale`.
- Never infer credentials. Only use the provided [SOURCE_METADATA].

To adapt this template, replace the placeholders with real data from your pipeline. [SOURCE_METADATA] should be a structured block containing the source's name, type, author, publication date, and any relevant credentials. [CLAIM_TOPIC] is the specific factual assertion being checked, and [CLAIM_DOMAIN] is the broader field (e.g., 'oncology,' 'monetary policy,' 'semiconductor manufacturing'). The OUTPUT_SCHEMA is designed for direct parsing by a downstream evidence-weighting module. The domain_mismatch_flags field is a critical safety mechanism; it forces the model to explicitly state when a source like a major newspaper might be incorrectly weighted as highly authoritative for a complex scientific claim, preventing a common and dangerous failure mode in automated fact-checking.

Before deploying this prompt, you must build a validation layer around the output. Check that the domain_authority_score is an integer between 1 and 5 and that the JSON schema is strictly valid. For high-stakes domains, route any assessment with a score of 3 or below, or any non-empty domain_mismatch_flags array, for human review. Your evaluation suite should include test cases where a high-credibility generalist source is assessed for a niche topic, expecting a low domain_authority_score and populated domain_mismatch_flags. This will catch the over-generalization failure mode before it affects your production verification results.

This prompt is designed to be called as a scoring function within a larger evidence-weighting pipeline. It should not be exposed directly to end users. Instead, it receives a structured claim domain and source metadata object, and returns a domain-match score with an explanation. The output is consumed by a downstream evidence ranker that combines this domain authority score with recency, directness, and corroboration signals to produce a final source weight for a specific claim. Treat this prompt as a deterministic component with probabilistic output: the same inputs should produce consistent scoring behavior, but the explanation text will vary across runs.

Integration pattern: Wrap the prompt in a thin service function that accepts [CLAIM_DOMAIN] (a controlled vocabulary string like clinical_trial_results or macroeconomic_forecast) and [SOURCE_METADATA] (a JSON object with fields: source_type, publisher, author_credentials, peer_review_status, publication_venue, methodology_disclosure, funding_disclosure). The service should validate inputs before calling the model: reject missing required fields, normalize the domain string against your taxonomy, and check that source_type is one of your enumerated values. If validation fails, return an error upstream rather than wasting a model call on malformed input.

Model choice and retries: Use a model with strong reasoning and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature=0 for scoring consistency. Implement a single retry with the same prompt if the output fails JSON schema validation or if the domain_match_score is missing. Do not retry more than once for the same input—log the failure and route to a human review queue. For high-throughput pipelines, batch up to 10 assessments per call by providing an array of source objects and requesting an array of assessments, but validate that the model returns exactly one assessment per input source.

Validation and logging: After receiving the model output, validate that domain_match_score is an integer between 0 and 100, that authority_tier matches your expected enum values, and that explanation is a non-empty string. Log every assessment with: input domain, source identifier, score, tier, explanation, model version, timestamp, and latency. This audit trail is essential for debugging score drift, detecting domain mismatch failures, and providing evidence for downstream verification reports. If the score exceeds a configurable threshold (e.g., 80), auto-accept the source for evidence matching. If below 30, auto-reject. Scores between 30 and 80 should trigger a secondary check or human review depending on claim risk level.

Failure modes to monitor: Watch for over-generalization where the model assigns high authority to a source outside its actual expertise (e.g., a Nobel laureate in physics scored highly for climate policy claims). Mitigate by including negative examples in the prompt's few-shot demonstrations. Watch for metadata hallucination where the explanation references credentials not present in the input—validate explanation text against input fields using a simple string-match check. Watch for score inflation on well-known brand names regardless of domain relevance—calibrate by periodically running a held-out test set of known domain-mismatch pairs and checking that scores remain low. Route any assessment where the explanation contradicts the score to human review.

Next steps: After deploying this prompt, build a calibration dashboard that plots domain-match scores against human-judged relevance ratings for a sample of 200+ source-domain pairs. Use this to tune your auto-accept and auto-reject thresholds. Version this prompt alongside your evidence ranker configuration so that changes to domain authority weighting are traceable and reversible. Do not use this prompt in isolation—always combine its output with recency, directness, and corroboration signals before making final source selection decisions.