Triage Threshold Assignment Prompt for Verification Pipelines

This prompt is designed for the moment when a verification pipeline moves from a single, hard-coded confidence threshold to a calibrated, domain-aware triage system. The ideal user is an operations engineer, a verification pipeline architect, or a product manager responsible for the cost, latency, and accuracy of a claim verification system. You need this prompt when you have historical pipeline performance data—such as precision and recall at various confidence levels, human review costs, and auto-verification error rates—and you must translate that data into a concrete routing configuration. The prompt does not score individual claims; it produces the threshold values and the business justification for them.

Before using this prompt, you must have already defined your claim domains, your evidence sufficiency scoring methodology, and your operational constraints. The prompt requires structured inputs: a description of the claim domain, the business risk tolerance (e.g., maximum acceptable false-positive rate for auto-verification), cost and latency targets for each routing path, and a summary of your scoring pipeline's historical performance. Without this context, the model cannot produce a meaningful trade-off analysis. Do not use this prompt if you are still prototyping your scoring function or if you lack performance data to calibrate against; in that case, start with a single conservative threshold and collect data before attempting a multi-tier configuration.

After running this prompt, you will receive a set of recommended thresholds, a sensitivity analysis showing how small threshold changes affect cost and accuracy, and a set of boundary-case tests to validate your configuration. The next step is to implement these thresholds in your routing logic, run the provided boundary-case tests against a golden dataset, and monitor the live system for threshold drift. Avoid the temptation to treat the output as a static configuration; schedule a re-run of this prompt whenever your scoring model is updated, your evidence sources change, or your business risk tolerance shifts.

This prompt template is designed for operations teams configuring automated verification pipelines. It takes a description of the verification domain, available evidence sources, and business constraints, and returns a recommended set of confidence thresholds with explicit justification. The output is meant to be used directly in routing logic or as a structured proposal for stakeholder review. Replace each square-bracket placeholder with your specific data before sending the prompt to the model.

textCopy
You are a verification pipeline architect. Your task is to recommend triage thresholds for an automated claim verification system. You will receive a description of the verification domain, the available evidence sources, and the business constraints that govern acceptable error rates, latency budgets, and review capacity.

## INPUT

**Verification Domain:** [DOMAIN_DESCRIPTION]
**Claim Types:** [CLAIM_CATEGORIES]
**Available Evidence Sources:** [EVIDENCE_SOURCES_WITH_AUTHORITY_TIERS]
**Business Constraints:**
- Maximum acceptable false-positive rate (auto-verified but incorrect): [MAX_FALSE_POSITIVE_RATE]
- Maximum acceptable false-negative rate (rejected but actually correct): [MAX_FALSE_NEGATIVE_RATE]
- Human review capacity per day: [REVIEW_CAPACITY]
- Target end-to-end latency per claim: [TARGET_LATENCY]
- Cost per human review: [REVIEW_COST]
- Cost per auto-verification error: [ERROR_COST]

**Current Confidence Scoring Behavior:**
- Confidence score range: [SCORE_RANGE]
- Observed score distribution: [SCORE_DISTRIBUTION_SUMMARY]
- Known failure modes: [FAILURE_MODES]

## OUTPUT SCHEMA

Return a JSON object with the following structure:

{
  "thresholds": {
    "auto_verify_min": <number>,
    "human_review_min": <number>,
    "reject_max": <number>
  },
  "expected_volumes": {
    "auto_verify_pct": <number>,
    "human_review_pct": <number>,
    "reject_pct": <number>
  },
  "justification": {
    "auto_verify_rationale": "<string>",
    "human_review_rationale": "<string>",
    "reject_rationale": "<string>",
    "trade_off_summary": "<string>"
  },
  "sensitivity_analysis": {
    "threshold_shift_impact": "<string>",
    "volume_spike_handling": "<string>",
    "evidence_quality_degradation": "<string>"
  },
  "boundary_cases": [
    {
      "scenario": "<string>",
      "confidence_score": <number>,
      "recommended_action": "auto_verify | human_review | reject",
      "reasoning": "<string>"
    }
  ],
  "implementation_notes": "<string>",
  "review_triggers": ["<string>"]
}

## CONSTRAINTS

1. Thresholds must be monotonically ordered: reject_max < human_review_min < auto_verify_min.
2. Expected volumes must sum to approximately 100%.
3. Justifications must reference specific business constraints, not generic best practices.
4. Sensitivity analysis must address at least: threshold shift by ±5 points, 2x volume spike, and one tier of evidence source degradation.
5. Include at least 3 boundary cases that test the edges of each threshold.
6. If human review capacity is insufficient for the expected volume at the recommended thresholds, flag this explicitly in implementation_notes.
7. Do not recommend thresholds that exceed the maximum acceptable error rates.

## RISK LEVEL

[RISK_LEVEL: low | medium | high | critical]

If RISK_LEVEL is high or critical, include a mandatory human-review checkpoint for any claim where the confidence score is within 5 points of a threshold boundary.

To adapt this template, start by filling in the domain-specific details under ## INPUT. The [DOMAIN_DESCRIPTION] should be a concise summary of what kinds of claims the pipeline verifies, such as "financial statement assertions in quarterly earnings reports." The [EVIDENCE_SOURCES_WITH_AUTHORITY_TIERS] field should list each source and its authority tier, for example "Tier 1: SEC filings, audited financials; Tier 2: reputable financial news; Tier 3: company press releases." The business constraints are critical: realistic numbers for error costs and review capacity will produce thresholds that can actually be operationalized. If you do not have observed score distribution data, run a calibration pass on a representative sample of claims first. The [RISK_LEVEL] placeholder controls whether the model adds boundary-case human-review requirements; set it to high or critical for regulated domains like healthcare or finance.

After receiving the model output, validate the JSON structure and check that the thresholds are monotonically ordered. Test the recommended thresholds against a held-out set of scored claims to verify that the expected volume percentages match reality. Pay special attention to the boundary cases: if the model's reasoning for a boundary case contradicts the threshold logic, that is a sign the thresholds need adjustment. For high-risk domains, route the threshold recommendations through a human approval step before deploying them to production routing logic. Store the threshold configuration, justification, and approval timestamp as part of your verification audit trail.

The Triage Threshold Assignment Prompt is designed to be called as a decision node within a larger verification pipeline, not as a standalone chat interaction. It receives a batch of scored claims, pipeline performance metrics, and business constraints, then returns a structured threshold recommendation. The prompt should be invoked after claim scoring is complete but before routing decisions are finalized. Typical integration points include a verification orchestrator that collects confidence scores from upstream models, aggregates them, and then calls this prompt to determine the cutoff values for auto-verify, human-review, and reject queues.

Wire the prompt into your application by constructing a request object that populates the [SCORED_CLAIMS_BATCH], [PIPELINE_METRICS], [BUSINESS_CONSTRAINTS], and [HISTORICAL_THRESHOLDS] placeholders. The [SCORED_CLAIMS_BATCH] should include claim IDs, confidence scores, evidence sufficiency ratings, and contradiction flags. The [PIPELINE_METRICS] must contain current accuracy, latency, and cost-per-claim figures. The [BUSINESS_CONSTRAINTS] should specify maximum acceptable error rates, review capacity limits, and cost budgets. After receiving the model's JSON output, validate it against the [OUTPUT_SCHEMA] before acting on the thresholds. Implement a validation layer that checks for required fields (auto_verify_threshold, human_review_threshold, reject_threshold), ensures threshold values are between 0 and 1, and confirms that auto_verify_threshold > human_review_threshold > reject_threshold. If validation fails, retry with the error message appended to the prompt context, up to a maximum of three attempts before escalating to a human operator.

For production deployments, log every threshold assignment decision with the full input context, model response, validation result, and final thresholds applied. This audit trail is essential for governance and for tuning the prompt over time. Use a structured logging format that captures the model version, prompt template version, timestamp, and pipeline run ID. When operating in regulated or high-stakes domains, always require human approval before applying new thresholds to live routing. Implement a review queue that presents the recommended thresholds alongside a sensitivity analysis showing how many additional claims would be auto-verified or escalated at adjacent threshold values. Avoid deploying threshold changes during peak verification loads; instead, schedule threshold updates during maintenance windows and run a shadow evaluation against a golden dataset of previously verified claims to detect regressions before the new thresholds affect live traffic.