Multi-Factor Evidence Weighting Prompt Template

The user question or information need that evidence is being scored against

What are the side effects of drug X in elderly patients?

Non-empty string check. Must contain a complete question or statement. Reject if only stop words or under 10 characters.

Array of retrieved evidence passages to score, each with text content and metadata

[{"id":"p1","text":"...","source":"PubMed","date":"2024-03"}]

Must be a valid JSON array. Each object requires id and text fields. Reject if array is empty or contains duplicate ids. Max 50 passages per batch.

Configurable weights for each scoring factor: relevance, recency, authority, specificity

{"relevance":0.40,"recency":0.20,"authority":0.25,"specificity":0.15}

Must be valid JSON object. All four keys required. Values must sum to 1.0 with tolerance of 0.01. Each weight must be between 0.0 and 1.0.

Maximum age in days before recency score begins to decay for time-sensitive queries

365

Must be a positive integer. Null allowed if query is not time-sensitive. If set, passages older than this threshold receive progressively lower recency scores.

Domain-specific authority indicators used to assess source credibility

["peer_reviewed","government_domain","citation_count","author_credentials"]

Must be a non-empty array of strings. Each signal must match a known authority indicator in the scoring rubric. Reject if signal names are unrecognized.

Expected JSON structure for scored passages including factor breakdowns and final weight

{"passage_id":"string","scores":{"relevance":0.0,"recency":0.0,"authority":0.0,"specificity":0.0},"weighted_total":0.0,"justification":"string"}

Must be valid JSON Schema or example structure. All four factor scores required. weighted_total must be computable from factor scores and weight config. justification field required for auditability.

Optional human-judged reference scores for calibration comparison

[{"passage_id":"p1","human_scores":{"relevance":0.85,"recency":0.70,"authority":0.90,"specificity":0.60}}]

Null allowed. If provided, must be valid JSON array matching passage ids in [PASSAGES]. Each human_score value must be between 0.0 and 1.0. Used for eval comparison, not prompt execution.

Every weighted score includes a per-factor breakdown with a specific reason tied to passage content

Output contains only a final aggregate score without factor-level justification or the justification is generic and untethered to the passage

Parse the JSON output and assert that each scored passage has a non-empty factor_breakdown object with at least relevance, recency, authority, and specificity keys, each containing a score and a non-empty rationale string

Model-assigned scores correlate with human expert scores at Spearman's ρ ≥ 0.7 on a held-out calibration set of 50 passage-query pairs

Model scores show weak or negative correlation with human judgments, or the model systematically over-scores low-quality passages

Run the prompt on a pre-labeled calibration dataset with 3 human annotators per passage; compute Spearman's rank correlation between median human score and model score; flag any passage where the absolute score difference exceeds 2 points on a 1-5 scale

Freshness scores reflect the temporal requirements of the query: time-sensitive queries penalize older passages, while evergreen queries do not apply inappropriate decay

A passage from last year is scored as low recency for an evergreen query, or a 3-year-old passage receives a high recency score for a breaking-news query

Construct a test set with 10 time-sensitive queries and 10 evergreen queries with known publication dates; assert that recency scores for passages older than the query's temporal window are below the configured staleness threshold for time-sensitive queries, and that recency scores for evergreen queries show no significant negative correlation with passage age

Specificity and relevance scores are independently assigned and do not collapse into a single dimension; vague passages that are on-topic receive high relevance but low specificity

A passage that is topically relevant but contains no concrete facts receives high scores on both relevance and specificity, indicating the model cannot distinguish general context from specific evidence

Include 5 vague-but-relevant passages in the test set; assert that specificity scores are at least 1 point lower than relevance scores on a 1-5 scale for these passages, and that the specificity rationale explicitly notes the absence of concrete claims or data

Authority scores are justified by observable source characteristics such as domain, author credentials, publication venue, or citation patterns, not by the passage's agreement with the query

The model assigns high authority to a passage simply because it supports the expected answer, or assigns low authority to a credible source without explanation

Audit a sample of 20 scored passages; for each, verify that the authority rationale references at least one source-level attribute such as domain, author, publisher, or peer-review status, and does not reference the passage's content alignment with the query

When confidence intervals are requested, the model expresses lower confidence for ambiguous or borderline passages and higher confidence for clear-cut assessments

The model assigns narrow confidence intervals to passages where factor signals conflict, or assigns wide intervals uniformly regardless of passage clarity

Construct a test set with 10 clear-cut passages where all factors align and 10 ambiguous passages where factors conflict; assert that the mean confidence interval width for ambiguous passages is at least 1.5x the mean width for clear-cut passages

The final aggregated score is not dominated by a single factor when multiple factors are configured with non-zero weights; changing one factor's weight produces a proportional change in the final score

One factor such as relevance consistently accounts for more than 80% of the final score variance even when other factors are weighted equally, indicating the model is ignoring the weighting configuration

Run the prompt on the same passage set with two different weight configurations where one factor's weight is doubled; assert that the final score changes in the expected direction for at least 80% of passages and that no single factor explains more than 60% of the score variance under equal weighting

The output is valid JSON matching the specified schema with all required fields present and correctly typed

The output is missing required fields, contains malformed JSON, uses incorrect types such as strings where numbers are expected, or includes extra unstructured text outside the JSON object

Validate the raw model output against the [OUTPUT_SCHEMA] using a JSON Schema validator; assert zero validation errors; retry once with a schema-reminder prompt if validation fails, then flag for human review on second failure