Reviewer Feedback Loop Prompt Template

This prompt is designed for verification operations teams and AI pipeline engineers who need to systematically capture reviewer corrections and convert them into actionable improvements for their automated triage system. The core job-to-be-done is closing the loop between human review decisions and the prompt logic that routes claims to auto-verification or human review. Without a structured feedback mechanism, triage systems drift: patterns of misclassification accumulate silently, and the same errors repeat across shifts and reviewers. This template produces a structured feedback record that captures the original triage decision, the reviewer's actual outcome, a discrepancy analysis, and suggested rule updates—making it possible to extract patterns and refine triage prompts over time.

Use this prompt when you have an established verification triage pipeline and a pool of human reviewers who regularly override or confirm automated routing decisions. The required inputs include the original claim text, the triage prompt's routing decision and confidence score, the reviewer's final verdict, and any reviewer notes explaining their reasoning. The prompt works best when reviewers are trained to document why they disagreed with the automated decision, not just what they changed. Do not use this template for real-time triage decisions or as a replacement for the primary triage prompt itself—it is a post-hoc analysis tool, not an operational routing prompt. It is also inappropriate for pipelines where reviewer throughput is so high that structured feedback capture would create unacceptable latency; in those cases, consider sampling-based feedback collection instead.

The output is a structured JSON record suitable for ingestion into a feedback database, analytics pipeline, or prompt refinement workflow. Each record includes explicit fields for discrepancy categorization (false positive auto-verification, missed escalation, correct routing with wrong confidence, etc.), suggested rule modifications, and a confidence delta that quantifies the gap between automated and human judgment. Before deploying this prompt, ensure your harness validates that feedback records are complete, that discrepancy categories are drawn from a controlled vocabulary, and that suggested rule updates reference specific triage prompt sections or conditions. The next step after implementing this feedback loop is to aggregate records over time, identify recurring failure patterns, and use those patterns to revise your triage prompt's classification logic, escalation thresholds, and evidence sufficiency rules.

This prompt template is designed to be integrated into a human review interface. When a reviewer overrides or corrects a prior triage decision, this prompt captures the original decision, the reviewer's outcome, and a structured analysis of the discrepancy. The output is a machine-readable feedback record that can be aggregated to identify patterns, refine triage prompts, and update routing rules. The template uses square-bracket placeholders for all dynamic inputs, ensuring it can be adapted to different verification domains and triage systems.

textCopy
You are a verification feedback analyst. Your task is to produce a structured feedback record from a reviewer's correction of an automated triage decision.

## INPUTS
- Original Claim: [CLAIM_TEXT]
- Claim Context: [CLAIM_CONTEXT]
- Auto-Triage Decision: [TRIAGE_DECISION]
- Auto-Triage Rationale: [TRIAGE_RATIONALE]
- Auto-Triage Confidence: [TRIAGE_CONFIDENCE]
- Reviewer Outcome: [REVIEWER_OUTCOME]
- Reviewer Notes: [REVIEWER_NOTES]
- Evidence Available: [EVIDENCE_SUMMARY]

## OUTPUT SCHEMA
Produce a JSON object with the following fields:
- `feedback_id`: A unique identifier for this feedback record.
- `discrepancy_type`: Classify the discrepancy as one of: "routing_error", "confidence_miscalibration", "evidence_misinterpretation", "claim_misclassification", "policy_violation", "no_discrepancy".
- `discrepancy_analysis`: A concise explanation of why the auto-triage decision differed from the reviewer outcome. Reference specific evidence or claim characteristics.
- `root_cause_category`: Classify the root cause as: "prompt_instruction_gap", "evidence_insufficiency", "domain_knowledge_gap", "threshold_misconfiguration", "ambiguous_claim", "reviewer_interpretation_difference".
- `suggested_rule_update`: A specific, actionable suggestion for updating triage prompts, routing rules, or confidence thresholds to prevent similar errors. If no update is needed, state why.
- `affected_triage_pattern`: Describe the pattern of claims that might be affected by this discrepancy (e.g., "claims with numerical comparisons across time periods", "claims involving legal terminology without cited statutes").
- `reviewer_action_taken`: Summarize the reviewer's action: "overrode_routing", "corrected_confidence", "changed_verdict", "requested_additional_evidence", "confirmed_auto_decision".
- `priority_for_review`: Assign a priority for incorporating this feedback: "critical", "high", "medium", "low". Base this on the potential impact of similar errors in production.

## CONSTRAINTS
- Do not include the reviewer's identity or any PII in the output.
- If the reviewer confirmed the auto-triage decision, set `discrepancy_type` to "no_discrepancy" and explain what worked correctly.
- The `suggested_rule_update` must be specific enough to be actionable by a prompt engineer. Avoid vague suggestions like "improve accuracy."
- Preserve uncertainty language from the reviewer's notes. Do not overstate confidence.
- If the reviewer identified missing evidence, note this in `discrepancy_analysis` and set `root_cause_category` to "evidence_insufficiency".

To adapt this template, replace each bracketed placeholder with data from your verification pipeline. The [CLAIM_TEXT] and [CLAIM_CONTEXT] fields should come from your claim extraction step. [TRIAGE_DECISION], [TRIAGE_RATIONALE], and [TRIAGE_CONFIDENCE] should be pulled from your triage prompt's structured output. [REVIEWER_OUTCOME] and [REVIEWER_NOTES] must be captured directly from the human review interface. The [EVIDENCE_SUMMARY] should include what evidence was available to the auto-triage system at decision time, not what the reviewer later found. This distinction is critical for accurate root cause analysis. For high-risk domains, ensure that the feedback record is stored immutably and linked to the original triage log for auditability.

The Reviewer Feedback Loop prompt is not a standalone tool; it is a structured logging step embedded in a human review workflow. When a reviewer overrides a triage decision, the application must capture the original triage output, the reviewer's corrected outcome, and the discrepancy context before calling this prompt. The prompt then produces a structured feedback record that can be stored, aggregated, and analyzed to refine the upstream triage prompt. The harness is responsible for ensuring that every human override generates exactly one feedback record and that no feedback is silently dropped.

Wire this prompt into the review interface's submission handler. When a reviewer submits a corrected verdict, the application should assemble the required inputs: the original claim text, the auto-triage decision with confidence score, the reviewer's outcome, and any reviewer notes. Validate these inputs before calling the model: the original triage decision must be a known enum value, the reviewer outcome must differ from the original decision (otherwise no feedback is needed), and reviewer notes must be non-empty for any override classified as a 'model error' rather than 'ambiguous claim.' After the model returns the structured feedback record, validate the output schema—particularly the discrepancy_category and suggested_rule_update fields—before writing to the feedback log. Reject records with hallucinated categories or updates that reference evidence not present in the input context.

Store validated feedback records in a queryable database with timestamps, reviewer identifiers, and the prompt version used. Run periodic aggregation jobs to extract patterns: which discrepancy_category values appear most frequently, which claim types generate the most overrides, and whether suggested_rule_update patterns converge on specific triage prompt improvements. Use these aggregations to propose concrete edits to the Verification Triage Decision Prompt Template, but never auto-apply feedback without human review of the proposed changes. Avoid the temptation to fine-tune on feedback records directly unless you have a statistically significant, deduplicated, and reviewer-quality-audited dataset—noisy reviewer feedback will degrade triage performance faster than a stable but imperfect prompt.