Mutation Testing Gap Analysis Prompt Template

This prompt is designed for QA engineers and test infrastructure leads who need to move beyond simple code coverage metrics and understand the quality of their test suite's assertions. The job-to-be-done is systematic weakness discovery: you have a set of mutation testing results showing which code mutations survived your test suite, and you need to categorize these survivors to identify patterns of ineffective assertions. This is not a prompt for generating new tests from scratch or for analyzing a single test failure. It is specifically for the analytical step that sits between running a mutation testing tool (like PIT, Stryker, or MutPy) and planning targeted test improvements.

The ideal user has already run mutation testing on a codebase and has a structured report of killed vs. survived mutations. The required context includes the source code diff for each surviving mutation, the test that was expected to kill it, and any failure or success output. Without this granular, per-mutation evidence, the prompt cannot produce a reliable gap analysis. The prompt works best when the mutation results are grouped by mutation operator type—such as conditionals boundary changes, arithmetic operator replacements, or return value modifications—because the analysis pattern relies on detecting systematic assertion weaknesses within each category. Do not use this prompt when you only have aggregate mutation scores without per-mutant detail, or when the goal is to explain a single flaky test rather than audit the test suite's overall assertion strength.

This prompt is a diagnostic tool, not a code generation engine. It produces a structured gap report that identifies categories of weakness, not a list of specific code fixes. The output should inform a human-led or agent-assisted test improvement sprint. Before using this prompt, ensure your mutation testing tool is configured to output per-mutant details in a parseable format. After receiving the gap analysis, the next step is to prioritize the identified weakness categories by risk impact and feed the highest-priority findings into a test generation workflow. Avoid using this prompt on mutation results that have not been deduplicated or on codebases where the test suite is known to be fundamentally broken—the analysis assumes a baseline of functional tests where assertion gaps are the primary concern, not missing test coverage entirely.

The following prompt template is designed to be copied directly into your AI harness, IDE, or testing pipeline. It accepts raw mutation testing output and relevant test suite context, then produces a structured gap report organized by mutation type. Every placeholder is enclosed in square brackets and must be replaced with real data before execution. The prompt is self-contained: it includes role assignment, output schema instructions, and explicit constraints that prevent the model from hallucinating fixes or making unsupported claims about test quality.

codeCopy
You are a senior QA engineer specializing in mutation testing and test suite effectiveness. Your task is to analyze the provided mutation testing results and test suite context to produce a structured gap report. The report must identify categories of mutations that survived the test suite, explain why they survived, and recommend concrete test improvement strategies.

## INPUT

### Mutation Testing Results
[MUTATION_RESULTS]

### Test Suite Context
- Test framework: [TEST_FRAMEWORK]
- Language: [LANGUAGE]
- Total tests: [TOTAL_TEST_COUNT]
- Test execution time: [EXECUTION_TIME]
- Coverage baseline (line/branch): [COVERAGE_BASELINE]
- Mutation operators applied: [MUTATION_OPERATORS_LIST]

### Repository Context (Optional)
[REPOSITORY_CONTEXT]

## OUTPUT SCHEMA

Return a JSON object with the following structure:

{
  "summary": {
    "total_mutations": <number>,
    "killed": <number>,
    "survived": <number>,
    "timeout": <number>,
    "mutation_score_percent": <number>,
    "analysis_timestamp": "<ISO 8601>"
  },
  "gap_categories": [
    {
      "mutation_type": "<string, e.g., CONDITIONALS_BOUNDARY, RETURN_VALUES, MATH, NEGATE_CONDITIONALS, VOID_METHOD_CALLS, INCREMENTS, INVERT_NEGS>",
      "survived_count": <number>,
      "severity": "<CRITICAL | HIGH | MEDIUM | LOW>",
      "description": "<explanation of why this mutation type survived and what testing weakness it reveals>",
      "affected_modules": ["<module path or class name>"],
      "root_cause": "<systematic reason: missing assertions, weak oracles, untested paths, mock overuse, etc.>",
      "improvement_strategies": [
        {
          "strategy": "<concrete action, e.g., Add boundary assertion for return value>",
          "effort": "<LOW | MEDIUM | HIGH>",
          "expected_impact": "<description of how many mutations this would kill>"
        }
      ]
    }
  ],
  "top_improvement_actions": [
    {
      "rank": <number>,
      "action": "<description>",
      "target_mutation_types": ["<string>"],
      "estimated_kill_increase": <number>,
      "effort": "<LOW | MEDIUM | HIGH>"
    }
  ],
  "limitations": [
    "<caveat about analysis, e.g., equivalent mutants not distinguished, timeout mutations may indicate performance issues rather than assertion gaps>"
  ]
}

## CONSTRAINTS

1. Only analyze mutations present in the provided results. Do not invent or assume mutations.
2. Group surviving mutations by mutation operator type. If fewer than 3 mutations of a type survived, merge them into an "OTHER" category with a note.
3. For each gap category, cite at least one specific example from [MUTATION_RESULTS] as evidence.
4. Severity assignment: CRITICAL if the mutation type reveals missing safety or security assertions; HIGH if core business logic is unverified; MEDIUM if error handling or edge cases are untested; LOW if cosmetic or unlikely to affect behavior.
5. Do not recommend deleting tests or lowering coverage targets.
6. If [REPOSITORY_CONTEXT] is provided, reference specific files or functions when suggesting improvements. If not provided, keep suggestions generic but actionable.
7. Mark any finding where you have low confidence with "confidence: LOW" and explain why.
8. Do not fabricate mutation scores, test names, or file paths not present in the input.

## EXAMPLES

### Example Input (Abbreviated)
Mutation: org.example.Calculator.add(int,int) - Replaced integer addition with subtraction - SURVIVED
Mutation: org.example.Calculator.divide(int,int) - Replaced double return with 0.0 - SURVIVED

### Example Output Fragment
{
  "gap_categories": [
    {
      "mutation_type": "MATH",
      "survived_count": 1,
      "severity": "HIGH",
      "description": "Arithmetic mutation in Calculator.add survived because the test only asserts non-null return, not the correct sum.",
      "affected_modules": ["org.example.Calculator"],
      "root_cause": "Weak assertion: test checks result is not null but does not verify the computed value.",
      "improvement_strategies": [
        {
          "strategy": "Add assertion verifying add(2,3) returns 5 exactly.",
          "effort": "LOW",
          "expected_impact": "Would kill all MATH mutations in Calculator.add"
        }
      ]
    }
  ]
}

## INSTRUCTIONS

1. Parse [MUTATION_RESULTS] to extract all surviving mutations.
2. Group by mutation operator.
3. For each group, determine the root cause of survival by comparing the mutation to what the existing tests actually assert.
4. Generate the output JSON following the schema exactly.
5. If the input is malformed or missing critical fields, return a JSON object with an "error" key describing what is missing.

To adapt this template, replace [MUTATION_RESULTS] with the raw output from your mutation testing tool—PIT, Stryker, Mull, or similar—in its native format. The prompt works best with detailed mutation descriptions that include the mutated source location, the operator applied, and the survival status. If your tool produces only summary statistics, add a preprocessing step that expands summaries into per-mutation records before passing them to the model. The [REPOSITORY_CONTEXT] placeholder is optional but strongly recommended: include relevant source files, test files, and any existing coverage reports to ground the analysis in real code. For high-risk codebases such as financial calculations, authentication logic, or safety-critical systems, always route the output through human review before acting on improvement recommendations. Validate the JSON output against the schema before ingestion into your test planning workflow, and log any schema violations for prompt refinement.

Integrating this prompt into a CI pipeline requires treating it as a post-mutation-testing analysis step, not a standalone script. The prompt expects structured mutation testing results—typically the output of tools like PIT, Stryker, or Mull—as its primary input. Before calling the LLM, your harness must parse raw mutation reports into the [MUTATION_REPORT] placeholder format, extracting fields like mutation operator, mutated class, line number, and kill status. This normalization step is critical because raw tool outputs vary in schema, and the prompt's reasoning quality depends on consistent input structure. The harness should also inject [CODEBASE_CONTEXT] by pulling relevant source files for surviving mutations, ensuring the model can reason about why a specific mutation evaded detection.

For production use, implement a validation layer that checks the LLM's output against a defined [OUTPUT_SCHEMA] before surfacing results. The schema should enforce that each gap category includes a mutation type, survival count, root cause hypothesis, and test improvement strategy. Use a JSON Schema validator in your pipeline to catch malformed responses and trigger a retry with the error message appended to the prompt. Set a maximum of two retries before falling back to a partial report with a warning flag. Log every LLM call with the input report hash, model version, response latency, and validation status for auditability. For high-risk codebases—safety-critical systems, financial logic, or security-sensitive modules—route the gap analysis through a human review queue before accepting test improvement recommendations. The harness should also compare gap categories across runs to detect regressions in test suite quality over time.

Model choice matters here: use a model with strong code reasoning capabilities and a large context window, as mutation reports for substantial codebases can be verbose. If the report exceeds the context limit, implement a chunking strategy that groups mutations by package or class and runs the prompt in parallel, then merge results with a deduplication step. Avoid running this analysis on every commit; instead, trigger it on a schedule (e.g., weekly) or after significant test suite changes. The harness should store historical gap reports to track whether identified gaps are being closed, creating a feedback loop that measures the prompt's real-world impact on test quality. Never treat the LLM's output as ground truth—always verify suggested test improvements by actually writing and running the tests against the surviving mutations.