Flaky Test Reproduction Step Generation Prompt

This prompt is for developers and QA engineers who have moved past initial triage and are stuck in the investigation phase. You have a flaky test failure log—stack traces, failure messages, environment metadata—but the failure refuses to reproduce with a simple ./gradlew test --tests FlakyTest or equivalent. The prompt's job is to generate a concrete, repeatable local reproduction script that maximizes the probability of triggering the intermittent failure. It is not a root cause analysis tool, a fix generator, or a test rewriter. Use it when you need to turn a ghost failure into a reproducible event you can debug.

The ideal input includes the full test failure log, the test code itself, any relevant configuration files (e.g., application.properties, CI environment variables), and a description of the conditions under which the failure was observed (CI worker type, concurrency level, test order, recent dependency changes). The prompt works best when the failure is suspected to be environmental, timing-related, or order-dependent. It is less effective for failures caused by truly random external service outages or one-off infrastructure blips that leave no trace in logs. Do not use this prompt when you have no failure evidence at all, or when the test failure is deterministic and already reproducible—in those cases, skip directly to debugging or fixing.

Before running the generated script, validate that it does not introduce destructive side effects (e.g., wiping a shared database, modifying production configs). The script should be treated as a diagnostic tool, not a permanent test fixture. After execution, compare the reproduction rate against the baseline failure rate in CI. If the script fails to reproduce the issue after the recommended number of iterations, you may need to gather additional context—such as thread dumps, database state snapshots, or network capture logs—and re-run the prompt with that expanded evidence. The next section provides the exact prompt template you can adapt and paste into your investigation workflow.

This prompt template is designed to ingest raw flaky test failure logs and produce a targeted, executable reproduction script. Its primary job is to translate intermittent failure signals—stack traces, timing data, environment variables, and test code—into a concrete local reproduction strategy. The template assumes you have already captured the failure context and are ready to generate a script that maximizes the probability of surfacing the defect on a developer machine or dedicated debugging environment.

textCopy
You are a test reliability engineer specializing in reproducing intermittent test failures. Your task is to generate a targeted reproduction script from the provided failure context. The script must maximize the probability of triggering the flaky failure locally.

## INPUT
- Failure Logs: [FAILURE_LOGS]
- Test Code: [TEST_CODE]
- Environment Context: [ENVIRONMENT_CONTEXT]
- Historical Flakiness Data (optional): [HISTORICAL_DATA]

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
{
  "reproduction_script": "string (a complete, runnable shell script or test runner command)",
  "environment_setup": ["list of prerequisite setup steps"],
  "execution_count": "integer (recommended number of repetitions)",
  "randomization_flags": ["list of flags to introduce non-determinism"],
  "stress_conditions": ["list of system conditions to simulate (e.g., CPU load, network latency)"],
  "key_observations": ["list of specific log patterns or metrics to monitor during reproduction"],
  "confidence_rationale": "string (explain why this strategy targets the hypothesized root cause)"
}

## CONSTRAINTS
- The reproduction script must be self-contained and idempotent.
- Do not suggest modifying production code or CI pipeline configuration.
- If the failure log suggests a race condition, include flags to randomize execution order or introduce artificial delays.
- If the failure log suggests an environmental dependency, include explicit version pinning and cleanup steps.
- Base all recommendations strictly on the provided logs and code. Do not invent failure modes not present in the evidence.
- If the evidence is insufficient to form a hypothesis, set "reproduction_script" to an empty string and explain the missing information in "confidence_rationale".

## EXAMPLES
[EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

To adapt this template, start by populating the [FAILURE_LOGS] placeholder with the raw output from your CI runner, including the full stack trace, test name, and any captured stdout/stderr. The [TEST_CODE] placeholder should contain the complete test function and any relevant helper or fixture code. For [ENVIRONMENT_CONTEXT], include details such as the operating system, language runtime version, container image digest, and any environment variables that differ between CI and local runs. The optional [HISTORICAL_DATA] field can accept a summary of past flakiness occurrences for this test, including timestamps and failure frequencies, which helps the model distinguish between a one-off glitch and a systemic pattern.

The [EXAMPLES] placeholder is critical for steering output quality. Provide one or two few-shot examples showing a failure log paired with a high-quality reproduction script. This teaches the model the expected level of detail in the stress_conditions and randomization_flags arrays. The [RISK_LEVEL] placeholder should be set to "low", "medium", or "high" based on the blast radius of the flaky test. For high-risk tests guarding critical release paths, the prompt should be followed by a human review step before the generated script is executed, and the output should be logged for auditability. After generating the script, always validate the JSON structure against the schema before passing it to an automated harness, and discard any output that invents root causes not grounded in the provided logs.

This prompt is designed to be called programmatically after a flaky test failure is detected in CI. The typical integration point is a post-failure hook in your test runner (e.g., Jest, Pytest, JUnit) or a webhook from your CI platform (GitHub Actions, Jenkins, Buildkite). When a test fails with a known flaky signature—intermittent pass/fail on retry, timeout variance, or non-deterministic assertion—the harness extracts the failure log, test code, and relevant environment metadata, then calls the LLM with this prompt to generate a local reproduction script. The output is not a fix; it is a diagnostic artifact that reduces the mean time to reproduce the failure locally.

The implementation harness should enforce a strict input contract before calling the model. Required inputs include: the full failure stack trace, the test source code (or at minimum the test method body), the CI execution command, and any environment variables that differ between CI and local defaults. If any required field is missing, the harness should return a structured error rather than calling the model with incomplete context—partial inputs produce unreliable reproduction scripts. After the model returns, validate the output against a schema that requires at minimum a reproduction_command string, a setup_steps array, and a confidence float between 0.0 and 1.0. Reject outputs that lack executable commands or that claim confidence above 0.9 without citing specific log evidence. For high-risk code paths (payment processing, auth, data deletion), route the generated reproduction script to a human for review before execution, and log the full prompt-response pair for audit.

Model choice matters here. Use a model with strong code generation and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature to 0.2 or lower to reduce variance in the reproduction logic. Enable JSON mode or structured outputs if your provider supports it, binding the response to a schema with reproduction_command, setup_steps, execution_count, randomization_flags, stress_conditions, expected_failure_signature, and confidence. Implement a retry loop with up to two retries if the output fails schema validation; on the second retry, append the validation error to the prompt as additional context. Log every generation attempt with the test name, CI run ID, model version, prompt hash, and validation result. This trace data becomes invaluable when debugging why a reproduction script didn't work—you can compare the generated steps against what actually reproduced the failure and feed that delta back into prompt improvements.