Test Case Difficulty Rating Prompt

ML engineers and QA leads use this prompt to move beyond flat pass/fail evaluation and understand which test cases expose the most risk. By assigning a difficulty rating to each example in a golden dataset—based on input ambiguity, required reasoning depth, and historical failure rates—teams can build weighted evaluation scoring and progressive testing gates. The output is a stratified difficulty label (easy, medium, hard, expert) that feeds directly into regression test harnesses, allowing you to prioritize fixes for the cases that matter most and avoid being flooded by failures on trivial examples.

This prompt is designed for offline dataset preparation, not real-time production routing. Run it as a batch job over your curated golden set before integrating the labels into your CI/CD evaluation pipeline. Each example should already have a known input and expected output; the prompt adds a difficulty annotation without modifying the test case itself. Wire the resulting labels into your eval harness to weight scores by difficulty, set per-tier pass thresholds, or gate promotion based on hard-case performance. For high-risk domains, always include a human review step to validate the assigned difficulty labels against domain expert judgment before the labels influence release decisions.

Do not use this prompt when the model must also generate the test case itself, or when you need real-time difficulty assessment during inference. It is not a routing classifier and should not gate live traffic. If your golden set is small or homogeneous, difficulty stratification may add complexity without value—stick to flat metrics until you have enough diversity to justify weighted scoring. After labeling, the next step is to integrate the difficulty field into your evaluation harness and monitor whether hard-case performance degrades across prompt versions.

This template assigns a difficulty rating to a single test case from your golden dataset. It evaluates the example across three axes—ambiguity, required reasoning depth, and historical failure rate—and produces a stratified label (e.g., EASY, MEDIUM, HARD, EXPERT) plus a confidence score. Use it to build weighted evaluation suites where harder examples carry more signal, or to implement progressive testing gates that only promote a prompt after it passes increasingly difficult tiers.

textCopy
You are a test suite calibration assistant. Your job is to rate the difficulty of a single test case from a golden dataset used to evaluate an AI system.

## INPUT
Test Case:
[TEST_CASE_INPUT]

Expected Output:
[EXPECTED_OUTPUT]

Domain Context:
[DOMAIN_CONTEXT]

Historical Failure Rate (if known):
[HISTORICAL_FAILURE_RATE]

## RATING CRITERIA
Rate the test case on three dimensions, each from 1 (lowest) to 5 (highest):

1. **Ambiguity**: How underspecified or open to interpretation is the input? A score of 1 means the input is precise and leaves little room for multiple valid answers. A score of 5 means the input is highly ambiguous and requires the model to make reasonable assumptions.

2. **Reasoning Depth**: How many inferential steps, domain knowledge applications, or logical operations are required to produce the expected output? A score of 1 means the answer is extractive or requires a single step. A score of 5 means the answer requires multi-step reasoning, synthesis across sources, or complex constraint satisfaction.

3. **Historical Failure Rate**: How often have previous prompt versions or models failed on this example? A score of 1 means the example is rarely failed. A score of 5 means the example is a known failure trigger. If no historical data is available, use [HISTORICAL_FAILURE_RATE] or state that you are estimating.

## OUTPUT SCHEMA
Return a single JSON object with this exact structure:
{
  "test_case_id": "[TEST_CASE_ID]",
  "difficulty_label": "EASY" | "MEDIUM" | "HARD" | "EXPERT",
  "confidence": 0.0-1.0,
  "dimension_scores": {
    "ambiguity": 1-5,
    "reasoning_depth": 1-5,
    "historical_failure_rate": 1-5
  },
  "rationale": "One-sentence explanation of the overall rating.",
  "failure_modes_expected": ["List of 1-3 likely failure modes for this example, or empty array if none are obvious."]
}

## DIFFICULTY LABEL RULES
- EASY: Average dimension score < 2.0
- MEDIUM: Average dimension score >= 2.0 and < 3.0
- HARD: Average dimension score >= 3.0 and < 4.0
- EXPERT: Average dimension score >= 4.0

## CONSTRAINTS
- Do not invent failure modes. Only list modes that are plausible given the input and expected output.
- If historical failure rate is unknown, set the score to 3 (neutral) and note this in the rationale.
- Confidence must reflect how certain you are about the label, not the model's likely performance.
- Return only the JSON object. No markdown fences, no commentary.

After pasting the template, replace every square-bracket placeholder with real data. [TEST_CASE_INPUT] and [EXPECTED_OUTPUT] are mandatory; [DOMAIN_CONTEXT] helps the model calibrate ambiguity and reasoning depth. If you lack historical failure data, set [HISTORICAL_FAILURE_RATE] to "unknown" and the model will default the score to 3. Wire the output into a post-processing step that validates the JSON schema, checks that the difficulty label matches the computed average, and logs any cases where confidence falls below 0.7 for human review. For high-stakes test suites, run this prompt across your entire golden set and stratify results before using them as weighted evaluation gates.

The difficulty rating prompt is not a one-off labeling tool—it is a calibration component inside a larger test suite management pipeline. When you wire it into an application, treat each rating as a structured artifact that must pass validation before it enters the golden dataset. The harness should accept a batch of test cases, call the prompt for each, validate the returned difficulty labels against the expected enum values (easy, medium, hard, extreme), and attach the rating plus the model's rationale to the test case metadata. Do not silently accept ratings that fall outside the enum or omit the required rationale field.

Build the harness with three layers: validation, retry, and review. Validation checks that the output is parseable JSON, that difficulty is one of the allowed enum values, and that rationale is a non-empty string. If validation fails, retry once with the same input plus the validation error message appended as a [CORRECTION_HINT]. If the retry also fails, flag the test case for human review rather than assigning a default difficulty. Log every rating decision with the model version, prompt version, input hash, and validation status so you can trace rating drift when the model or prompt changes. For high-stakes test suites where difficulty ratings gate release decisions, require a human reviewer to spot-check a stratified sample—at least 10% of cases from each difficulty tier—before the ratings are committed to the golden dataset.

Model choice matters here. Use a model with strong reasoning capabilities for difficulty rating, as the task requires judging ambiguity, reasoning depth, and historical failure patterns. Avoid lightweight or speed-optimized models that may produce inconsistent ratings across similar cases. If you are rating hundreds of test cases, batch them in groups of 20–30 and shuffle the order to reduce position bias. Store the ratings alongside the test case in your dataset management system with a schema that includes difficulty, rationale, rater_model, rater_prompt_version, rating_timestamp, and human_reviewed fields. This schema lets you later query for rating consistency, identify cases where human reviewers disagreed with the model, and recalibrate the prompt when rating distributions drift. The next step after wiring this harness is to run a calibration pass: rate a known set of cases, compare against human-assigned difficulty labels, and tune the prompt's [DIFFICULTY_CRITERIA] placeholder until inter-rater agreement reaches your target threshold.