Evidence Strength Annotation Prompt for Review Items

This prompt is designed for compliance engineers and operations leads who need to annotate the strength of evidence attached to review items before a human reviewer sees them. The goal is to produce a structured assessment of each source's reliability, identify gaps in the evidence chain, and flag items where the supporting material is too weak to support a confident decision. Use this prompt when you are building a human review queue and need to pre-process evidence so reviewers can quickly assess source reliability without reading every document from scratch.

This prompt belongs in a pre-review pipeline stage, after evidence has been collected and attached to a review item but before the item is routed to a human reviewer. It is not a replacement for human judgment on evidence weight; it is a triage aid that surfaces reliability concerns early. The ideal input is a review item with a clear decision question and a set of attached sources. The output should be a structured annotation per source, including a reliability rating, a justification grounded in the source material, and a list of identified evidence gaps. Do not use this prompt when the evidence is already highly structured and machine-verifiable, or when the review decision is purely procedural and does not require source credibility assessment.

Before deploying, define what 'weak evidence' means in your domain. For a financial audit, a verbal confirmation without a paper trail might be weak. For a policy violation review, a single anonymous report might be insufficient. Codify these rules in the [CONSTRAINTS] placeholder so the model applies consistent standards. Pair this prompt with an evaluation step that checks whether the model correctly identified a known-weak source in a golden dataset, and whether it missed any gaps that a human auditor later found. If the prompt will run on high-stakes items, route outputs where the model flags 'insufficient evidence' for mandatory human review before any downstream action is taken.

This prompt template is designed to be copied directly into your application or testing environment. It instructs the model to examine each piece of evidence associated with a review item, assign a strength label, flag reliability concerns, and identify gaps where evidence is missing or insufficient. Replace every square-bracket placeholder with your actual data before sending the prompt to the model. The template assumes you have already assembled the review item content, its supporting evidence, and any domain-specific annotation rules.

codeCopy
You are an evidence quality annotator for a human review queue. Your job is to examine each piece of evidence attached to a review item and produce a structured annotation that helps a human reviewer quickly assess source reliability and evidence completeness.

## INPUT

Review Item:
[REVIEW_ITEM_TEXT]

Evidence List:
[EVIDENCE_LIST]

Annotation Rules:
[ANNOTATION_RULES]

## OUTPUT SCHEMA

Return a JSON object with the following structure:
{
  "item_id": "[ITEM_ID]",
  "annotations": [
    {
      "evidence_id": "string",
      "strength_label": "strong" | "moderate" | "weak" | "insufficient",
      "reliability_flags": ["string"],
      "gap_notes": "string or null",
      "rationale": "string"
    }
  ],
  "overall_assessment": {
    "strongest_evidence_id": "string or null",
    "critical_gaps": ["string"],
    "reviewer_guidance": "string"
  }
}

## CONSTRAINTS

- Assign `strength_label` based on source credibility, directness of evidence, and corroboration.
- Use `reliability_flags` to surface concerns such as hearsay, outdated source, conflicting statements, anonymous source, or single-source dependency.
- Populate `gap_notes` when evidence is missing, incomplete, or requires additional verification. Use null if no gaps are identified.
- The `rationale` field must explain why the strength label was assigned in one to two sentences.
- `critical_gaps` in the overall assessment should list only gaps that would prevent a reviewer from making a confident decision.
- Do not invent evidence. If the evidence list is empty, mark all annotations as `insufficient` and explain in `reviewer_guidance`.
- If [RISK_LEVEL] is `high`, flag any single-source or uncorroborated evidence as `weak` regardless of apparent credibility.

## EXAMPLES

[FEW_SHOT_EXAMPLES]

## OUTPUT

Return only the JSON object. No additional commentary.

Adaptation guidance: Replace [REVIEW_ITEM_TEXT] with the full content of the item under review. [EVIDENCE_LIST] should contain each evidence record with its identifier, source description, date, and content. [ANNOTATION_RULES] should encode your organization's specific criteria for strength labels, such as regulatory standards, internal policy thresholds, or domain-specific reliability heuristics. [RISK_LEVEL] controls whether the prompt applies stricter evidence requirements for high-risk review contexts. [FEW_SHOT_EXAMPLES] should include two to four annotated examples that demonstrate correct label assignment, especially for borderline cases where moderate versus weak distinction matters. Test this prompt with at least ten review items where you already know the expected annotations before deploying to production. If your evidence list is large, consider batching or pre-filtering to keep the prompt within context limits while preserving all evidence that could affect the annotation.

The Evidence Strength Annotation prompt is designed to be called as a pre-processing step before a review item reaches a human reviewer. In a typical implementation, a review queue service receives a new item, extracts the source list and any associated evidence snippets, and calls the LLM with this prompt to annotate each source's strength. The output is a structured JSON payload that gets attached to the review item as metadata, allowing the reviewer interface to render strength badges, sort sources by reliability, and highlight gaps before the human spends time on the item.

Wire this prompt into an asynchronous pipeline, not a synchronous user-facing request. After the review item is created, enqueue an annotation job. The job fetches the item's sources, hydrates the [SOURCES] placeholder with an array of objects containing source_id, source_type, and evidence_snippet, and populates [DOMAIN] with the review domain (e.g., 'financial_transaction', 'clinical_note', 'security_alert'). On success, parse the JSON output and validate it against a strict schema: each source must have a strength_label from the allowed enum (STRONG, MODERATE, WEAK, INSUFFICIENT), a rationale string, and a gaps array. Reject any output where a source from the input is missing or an unknown source appears. If validation fails, retry once with the validation error injected into [CONSTRAINTS] as a repair instruction. If the retry also fails, flag the item for manual annotation and log the raw output for debugging.

For high-stakes domains like compliance or clinical review, never surface unvalidated annotations directly to reviewers. Implement a confidence gate: if the model's output includes a WEAK or INSUFFICIENT label, the reviewer UI should prominently flag that source and may require the reviewer to confirm or override the annotation before closing the item. Log every annotation event—input sources, model output, validation result, and any human override—to an audit table. This creates a feedback loop for evaluating annotation consistency over time. Use the eval checks described in the playbook (annotation consistency, missed weak-evidence flags) as periodic batch jobs that sample annotated items and compare model labels against human reviewer overrides, surfacing drift or systematic errors before they affect downstream decisions.