Domain-Adapted Evidence Weighting Prompt Template

The user question or information need that evidence is being weighted against.

What are the side effects of JAK2 inhibitors in elderly patients with renal impairment?

Check that the query is a non-empty string. For domain-specific workflows, verify that the query contains at least one domain term from the configured terminology list. Reject queries that are purely conversational or off-topic.

An array of retrieved passages to be weighted. Each passage should include text content and available metadata.

[{"id": "doc_1", "text": "...", "source": "PubMed", "date": "2024-03-15", "authority": "peer-reviewed"}]

Validate that the array is non-empty and each passage object contains a non-empty text field. Check that metadata fields used by the domain weighting rules such as source, date, and authority are present. If a required metadata field is missing, flag the passage for pre-processing or exclusion before scoring.

A configuration object defining domain-specific weighting rules, terminology, authority signals, and evidence standards.

{"domain": "clinical_medicine", "authority_tiers": {"peer-reviewed": 1.0, "guideline": 0.9, "expert_opinion": 0.6}, "recency_threshold_days": 730, "required_metadata": ["source", "date"]}

Validate that the domain_config object contains a non-empty domain identifier and at least one weighting rule. Check that authority_tiers values are numeric and between 0 and 1. Verify that recency_threshold_days is a positive integer. Reject configurations that reference undefined domain identifiers or missing rule parameters.

The expected JSON schema for the weighted evidence output, including score fields, justification requirements, and confidence indicators.

{"type": "object", "properties": {"passage_id": {"type": "string"}, "weighted_score": {"type": "number"}, "factor_breakdown": {"type": "object"}, "justification": {"type": "string"}}, "required": ["passage_id", "weighted_score", "factor_breakdown"]}

Validate that the schema is valid JSON Schema syntax. Check that required fields include passage_id, weighted_score, and at least one factor breakdown or justification field. Ensure numeric fields have defined minimum and maximum constraints. Reject schemas that allow unbounded score ranges.

Operational constraints for the weighting task, including score ranges, factor weight limits, and output formatting rules.

{"score_range": [0, 1], "max_factors": 5, "require_justification": true, "confidence_interval": true, "max_passages": 20}

Check that score_range is a two-element array with min less than max. Verify that max_factors is a positive integer not exceeding 10. If require_justification is true, confirm that the output schema includes a justification field. Validate that max_passages does not exceed the model context window capacity for the expected passage length.

Optional array of human-annotated reference examples for calibrating the model's weighting behavior against domain expert judgments.

[{"passage_id": "doc_1", "human_score": 0.85, "human_factors": {"relevance": 0.9, "authority": 0.8, "recency": 0.7}}]

If provided, validate that each reference contains a passage_id matching an entry in evidence_passages, a numeric human_score in the configured score_range, and a human_factors object with keys matching the domain_config factor names. Flag references with scores outside the configured range. This field is optional; set to null when calibration data is unavailable.

A list of domain-specific terms, synonyms, and entity patterns that the weighting model should recognize when assessing relevance and specificity.

["JAK2 inhibitor", "myelofibrosis", "ruxolitinib", "fedratinib", "renal impairment", "eGFR", "CKD stage"]

Validate that the list is non-empty and each term is a non-empty string. Check for duplicate terms. For clinical domains, verify that terms do not include PHI or patient identifiers. For legal domains, verify that terms do not include privileged case references. Flag terms that appear to be generic stop words rather than domain-significant vocabulary.

Rules for excluding passages from scoring based on domain-inappropriate content, outdated information, or prohibited source types.

{"exclude_sources": ["social_media", "unverified_preprint"], "exclude_before_date": "2010-01-01", "exclude_if_missing": ["source", "date"], "exclude_content_patterns": ["case report with n=1"]}

Validate that exclude_sources contains only strings matching known source type categories. Check that exclude_before_date is a valid ISO date string. Verify that exclude_if_missing fields exist in the evidence passage metadata schema. Test exclusion rules against a sample passage to confirm they execute without errors before running the full batch.

Prompt correctly identifies and weights domain-specific authority markers (e.g., court level for legal, journal tier for medical, regulatory body for finance) in at least 90% of test cases

Generic authority signals (e.g., domain age, backlinks) dominate domain-specific signals; peer-reviewed journal weighted same as blog

Run 20 domain-specific passage pairs with known authority differentials; measure whether domain-appropriate source ranks higher

Prompt recognizes domain synonyms, acronyms, and technical terms as equivalent to query intent; penalizes passages using correct terms in wrong context

Surface-level keyword match dominates; passage with correct domain term used incorrectly scores higher than passage with synonym used correctly

Curate 15 query-passage pairs with controlled terminology traps; measure precision of relevance ranking against domain-expert labels

Prompt applies domain-appropriate evidence standards (e.g., peer-review requirement for medical claims, primary source preference for legal) in scoring decisions

Anecdotal or low-quality evidence scores above domain-standard evidence; preprint weighted equal to published RCT

Construct 10 evidence pairs where one meets domain standard and one does not; verify standard-compliant passage receives higher score

Prompt refuses or flags low confidence when domain-adapted criteria are applied to queries outside the configured domain

Prompt confidently applies medical evidence standards to a legal query or vice versa without flagging domain mismatch

Submit 10 out-of-domain queries; check for explicit domain-mismatch flag or low-confidence signal in output

Every weighted score includes a traceable justification referencing specific passage content and the domain criterion applied

Scores appear without justification; justifications are generic (e.g., 'relevant') without citing passage content or domain rule

Parse output for justification field per score; verify each justification references a specific passage span and names the domain criterion used

Prompt applies domain-appropriate recency decay (e.g., rapid decay for drug interaction evidence, slow decay for constitutional law) when [RECENCY_WEIGHT] is configured

All domains receive identical recency treatment; 10-year-old legal precedent penalized same as 10-year-old medical trial

Run 5 time-sensitive queries per supported domain with known stale vs. fresh evidence pairs; verify domain-appropriate decay

Prompt outputs confidence levels that correlate with evidence ambiguity; low confidence when sources conflict or evidence is thin

High confidence assigned to scores derived from single weak source or conflicting evidence set

Measure confidence scores against human-rated ambiguity across 30 passages; expect confidence-ambiguity correlation above 0.7

Output matches [OUTPUT_SCHEMA] exactly: all required fields present, types correct, no extra fields

Missing required fields; score field contains string instead of number; justification field absent or null when required

Validate output against [OUTPUT_SCHEMA] using JSON Schema validator; run across 50 varied inputs and require 100% structural compliance