Prompt Output Regression Severity Scoring Prompt

This prompt is for triage engineers, QA leads, and product managers who need to prioritize a growing queue of regression failures after a prompt change or model update. Its job is to grade each regression failure on a structured severity scale, considering user impact, failure frequency, data corruption risk, and reversibility. The output is a prioritized remediation queue with severity justifications, enabling the team to focus on the most damaging regressions first instead of treating every test failure as equally urgent.

Use this prompt when you have a batch of regression test failures from a golden dataset run and need to decide what to fix before a release. It is designed for production prompt pipelines where a change has already been tested, failures have been identified, and the next step is operational triage. The prompt expects a list of failure records as input, each containing the test case ID, the input, the expected output, the actual output, and any existing failure category tags. It works best when paired with the Regression Failure Categorization Prompt Template, which can pre-classify failures before severity scoring.

Do not use this prompt as a real-time production guardrail or for scoring failures in live user traffic—it is designed for offline evaluation suites, not streaming inference. It is also not a substitute for human judgment in safety-critical domains. If a regression involves regulated outputs, financial data, or clinical content, the severity score should be reviewed by a domain expert before remediation actions are taken. The prompt produces a structured JSON output, making it straightforward to wire into a CI/CD pipeline or a triage dashboard.

This prompt template is designed to be dropped into a triage workflow where a regression test suite has already produced a list of failures. It expects each failure to include the input that triggered it, the expected output, the actual output, and any available metadata such as the test case ID or category. The model's job is not to re-evaluate whether the failure is valid—that should already be established—but to assess how urgently each failure needs to be addressed based on user impact, data integrity risk, reversibility, and blast radius.

textCopy
You are a regression severity scorer for an AI product triage team. Your job is to review a list of confirmed regression failures and assign each one a severity score with clear justification. Do not re-judge whether the failure is valid. Assume each listed failure has already been confirmed as a regression.

## INPUT
You will receive a list of regression failures. Each failure includes:
- test_case_id: unique identifier for the test case
- input: the prompt or query that triggered the failure
- expected_output: the previously accepted correct output
- actual_output: the current output that deviates from expected
- category: the failure category (e.g., hallucination, format_drift, over_refusal, instruction_misinterpretation, context_mishandling)
- affected_user_segment: who is impacted (e.g., all_users, enterprise_only, free_tier, internal_tools)
- frequency_estimate: how often this failure is expected to occur (e.g., rare, occasional, frequent, constant)

[FAILURE_LIST]

## SEVERITY SCALE
Use this 4-level scale:
- **P0 - Critical**: Causes data corruption, irreversible harm, safety violation, or complete system unusability for a broad user segment. Requires immediate rollback or hotfix.
- **P1 - High**: Significantly degrades core functionality, produces materially wrong answers that could cause user harm or business loss, or affects a large user segment with no workaround. Fix within 24 hours.
- **P2 - Medium**: Noticeable quality degradation, incorrect but not harmful outputs, or impact limited to a narrow user segment or edge case. Fix in the next scheduled release.
- **P3 - Low**: Cosmetic issues, minor format deviations, rare edge cases with minimal user impact, or failures that are technically regressions but produce acceptable outputs. Fix when convenient.

## SCORING CRITERIA
For each failure, consider:
1. **User Impact**: How severely does this affect the end user's experience or trust? Can they still accomplish their task?
2. **Data Integrity Risk**: Could this failure corrupt stored data, propagate bad information downstream, or cause cascading errors?
3. **Reversibility**: Can the harm be undone? Is the output ephemeral or persisted?
4. **Blast Radius**: How many users, use cases, or downstream systems are affected?
5. **Workaround Availability**: Is there a reasonable alternative path for users?
6. **Category Severity Weight**: Certain failure categories carry higher inherent risk. Hallucination in regulated domains and data corruption should be weighted higher than format drift or cosmetic issues.

## OUTPUT FORMAT
Return a JSON object with this exact structure:
{
  "severity_summary": {
    "total_failures": <number>,
    "p0_count": <number>,
    "p1_count": <number>,
    "p2_count": <number>,
    "p3_count": <number>
  },
  "prioritized_queue": [
    {
      "rank": <number starting from 1>,
      "test_case_id": "<string>",
      "severity": "P0|P1|P2|P3",
      "justification": "<2-4 sentence explanation referencing specific scoring criteria>",
      "recommended_action": "<one of: immediate_rollback, hotfix, next_release, backlog>"
    }
  ],
  "triage_notes": "<optional overall assessment of the regression batch, patterns noticed, or escalation recommendations>"
}

## CONSTRAINTS
- Every failure in the input must appear in the prioritized_queue exactly once.
- Rankings must be strict: no ties. If two failures seem equally severe, break the tie by prioritizing the one with broader user impact.
- Justifications must reference specific scoring criteria, not generic statements.
- If any failure involves potential data corruption, safety risk, or regulated-domain hallucination, flag it in triage_notes for immediate human review regardless of assigned severity.
- Do not invent failures or modify the input data.

[OUTPUT_SCHEMA]

## EXAMPLES
[EXAMPLES]

Adaptation guidance: Replace [FAILURE_LIST] with your actual regression failure data, structured as described in the INPUT section. The [OUTPUT_SCHEMA] placeholder can be removed if your application layer already enforces JSON schema validation—the inline format specification is sufficient for most models. The [EXAMPLES] placeholder should be populated with 2-3 scored examples showing different severity levels, especially edge cases where P1 and P2 are hard to distinguish. If your domain has specific severity definitions (e.g., SOC2 compliance impact, SLA penalties), add those as additional scoring criteria rather than replacing the existing ones. For high-stakes domains like healthcare or finance, add a [RISK_LEVEL] constraint that forces P0 classification for any failure touching protected data or regulated outputs.

This prompt is designed to sit after a regression test suite runner has produced a list of failing test cases. The harness should collect each failure's metadata—input, expected output, actual output, failure category, and any existing severity tags—and feed them into the prompt as structured [FAILURE_RECORDS]. The prompt is not a real-time classifier; it is a batch triage tool meant to run once per test run or per release candidate. Expect latency of several seconds per batch, and design your pipeline to call this prompt asynchronously after the test suite completes.

Input assembly: Build the [FAILURE_RECORDS] array by joining the regression runner's output with any production incident data. Each record must include at minimum: failure_id, test_case_name, expected_behavior, actual_behavior, failure_category, and affected_user_path. If you have production telemetry, add user_impact_estimate and frequency_last_7d fields. The prompt's severity scale depends on these fields being populated; missing data will produce low-confidence scores. Validate the input array before sending—reject records with null actual_behavior or missing failure_category and route them to a manual triage queue.

Output validation: The prompt returns a JSON array of severity assessments. Before accepting the output, validate that every record has a severity_score (integer 1-5), a severity_label matching the defined scale, and a non-empty justification. Reject the entire batch if more than 5% of records fail validation and retry once with a stricter [CONSTRAINTS] block that emphasizes completeness. If the retry also fails, escalate the batch to a human triage lead. Log every validation failure with the raw model output for debugging.

Human review gate: Severity scores of 4 (high) or 5 (critical) should trigger a human review step before the remediation queue is finalized. The harness should post these high-severity failures to a review channel (Slack, Jira, or a custom dashboard) with the prompt's justification and a link to the full failure record. Do not auto-merge critical severity items into a remediation sprint without human sign-off. For lower severities (1-3), the harness can auto-populate the triage backlog with the prompt's remediation_priority field as the initial sort key.

Model choice and retries: Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature=0 or very low (0.1) to maximize scoring consistency across runs. Implement a single retry on format validation failure with an explicit error message appended to the [CONSTRAINTS] block. Do not retry on content disagreements—those belong in the human review step. Log the model version, prompt version, and input hash with every run for auditability.

Avoid wiring this prompt directly into an automated rollback decision. Severity scores inform prioritization, not automated release gates. A critical severity finding should pause a rollout and notify the on-call engineer, but the decision to roll back must remain human-driven. The prompt's justification field is designed to accelerate that decision, not replace it.