Test Order Dependency Detection Prompt

This prompt is designed for platform engineers and QA automation leads who need to identify test pollution caused by execution order. The core job-to-be-done is analyzing test execution logs from both randomized and sequential runs to surface dependency pairs—tests that pass in isolation but fail when run after a specific predecessor. The ideal user has access to CI logs with timestamps, test outcome data, and ideally, execution order metadata from test runners that support shuffling (e.g., pytest-randomly, rspec --order random). You should use this prompt when you have evidence of order-dependent failures (a test that fails only in certain CI shards or after a full suite run) but haven't yet isolated the contaminating predecessor. The prompt requires structured input: a list of test cases with their outcomes across multiple runs, the execution order for each run, and any relevant failure messages or stack traces.

Do not use this prompt when you lack execution order data or when failures are clearly caused by external factors like network timeouts, resource exhaustion, or infrastructure flakiness. This prompt is not a general flaky test classifier—it specifically targets state leakage between tests (e.g., shared mutable state, uncleaned database records, modified environment variables, or global singleton mutations). If your test framework doesn't randomize order, you'll need to first collect data from at least one randomized run and one sequential run to provide the necessary contrast. The prompt also assumes tests are independent by design; if your suite intentionally relies on ordered execution, this analysis will produce false positives. For high-risk codebases (e.g., financial transaction processing, healthcare data pipelines), always pair the prompt's output with a manual review of the identified dependency pairs before refactoring, as the prompt may miss subtle contamination through indirect side effects or external services.

Before using this prompt, gather your test execution data into a structured format: each run should include the test name, outcome (pass/fail/error), duration, and the ordered list of tests executed before it. If you have multiple CI runs, include at least one randomized run and one sequential run to provide the contrast needed for dependency detection. The prompt will output dependency pairs with supporting evidence, but you should validate each pair by attempting to reproduce the failure locally with the suspected predecessor. After receiving the output, prioritize remediation by focusing on tests that contaminate multiple downstream tests or that affect critical path coverage. Avoid the temptation to simply reorder tests as a fix—this masks the underlying state leakage and will cause failures to resurface unpredictably.

The following prompt template is designed to analyze test execution logs from both randomized and sequential runs to identify test order dependency (TOD) pairs. TOD occurs when a test passes in isolation but fails when run after a specific predecessor due to shared mutable state, unreset singletons, or leaked side effects. This template expects structured log input and produces a dependency pair report with evidence, confidence levels, and remediation suggestions. Use it as the core analysis step in a broader pipeline that includes log collection, randomization, and human review of high-confidence findings.

textCopy
You are a test reliability engineer analyzing test execution logs to detect test order dependencies (TOD). A TOD exists when Test B passes in isolation or randomized runs but consistently fails when executed immediately after Test A. This indicates Test A is polluting shared state that Test B depends on.

## INPUT

[TEST_EXECUTION_LOGS]

Provide execution logs in the following structure:
- Randomized run results: multiple runs with randomized test order, including pass/fail status, duration, and failure messages per test.
- Sequential run results: at least one run with a fixed, known order, including pass/fail status, duration, and failure messages per test.
- Test code snippets for any failing tests (optional but recommended for higher confidence).
- Shared resource inventory: list of known shared resources (databases, files, environment variables, singletons, caches) accessible to the test suite.

## OUTPUT_SCHEMA

Return a JSON object with the following structure:
{
  "dependency_pairs": [
    {
      "polluter_test": "string (test that causes the failure)",
      "victim_test": "string (test that fails due to pollution)",
      "confidence": "high | medium | low",
      "evidence": {
        "randomized_runs": "string (summary of pass/fail pattern in randomized runs)",
        "sequential_runs": "string (summary of pass/fail pattern in sequential runs)",
        "failure_signature": "string (consistent error message or exception type)",
        "shared_resource_suspected": "string (which shared resource is likely polluted)"
      },
      "remediation": "string (specific cleanup, isolation, or ordering fix suggestion)",
      "requires_human_review": true
    }
  ],
  "no_dependency_found": false,
  "analysis_notes": "string (summary of overall findings, limitations, and recommendations for further investigation)"
}

## CONSTRAINTS

- Only report a dependency pair when there is evidence from both randomized and sequential runs. A failure in sequential runs alone is insufficient; you must observe the test passing in randomized runs where the suspected polluter runs after the victim or not at all.
- Confidence must be "high" only when the failure is reproducible across multiple randomized and sequential runs with the same polluter-victim ordering and the same failure signature.
- If no dependency pairs meet the evidence threshold, set "no_dependency_found" to true and explain why in "analysis_notes".
- Do not speculate about code you cannot see. If test code snippets are not provided, note this limitation in "analysis_notes" and reduce confidence accordingly.
- Flag all findings with "requires_human_review": true. This analysis is a starting point for investigation, not a final determination.

## EXAMPLES

Example input (abbreviated):
Randomized run 1: TestA PASS, TestC PASS, TestB FAIL (NullPointerException at line 42)
Randomized run 2: TestC PASS, TestB PASS, TestA PASS
Sequential run: TestA PASS, TestB FAIL (NullPointerException at line 42), TestC PASS
Shared resources: DatabaseConnectionPool (singleton), /tmp/test_uploads/ directory

Example output:
{
  "dependency_pairs": [
    {
      "polluter_test": "TestA",
      "victim_test": "TestB",
      "confidence": "high",
      "evidence": {
        "randomized_runs": "TestB failed in 3 of 4 randomized runs, always when TestA executed immediately before it. TestB passed in the one run where TestA ran after TestB.",
        "sequential_runs": "TestB consistently fails in sequential runs where TestA precedes it.",
        "failure_signature": "NullPointerException at DatabaseConnectionPool.getConnection line 42",
        "shared_resource_suspected": "DatabaseConnectionPool singleton not reset after TestA"
      },
      "remediation": "Add @AfterEach cleanup in TestA to reset DatabaseConnectionPool to its initial state, or use a test-specific connection pool instance.",
      "requires_human_review": true
    }
  ],
  "no_dependency_found": false,
  "analysis_notes": "One high-confidence dependency pair identified. Recommend reviewing TestA's teardown logic and TestB's connection pool assumptions."
}

## RISK_LEVEL

[RISK_LEVEL]

If RISK_LEVEL is "high" (e.g., tests cover payment processing, authentication, or data integrity):
- Require human review of all findings before any code changes.
- Do not suggest automated fixes; only suggest investigation paths.
- Flag any finding where the failure signature involves data corruption or security-sensitive paths.

If RISK_LEVEL is "medium" or "low":
- Still require human review but allow broader remediation suggestions.
- Include specific code patterns for cleanup if test code is provided.

To adapt this template for your environment, replace [TEST_EXECUTION_LOGS] with your actual log data in the specified structure. If your test framework produces logs in a different format, preprocess them into the expected structure before passing them to the model. The [RISK_LEVEL] placeholder should be set based on the domain of the tests under analysis—use "high" for security, financial, or data integrity tests, and "medium" or "low" for less critical paths. If you have access to test source code, include it in the input to improve confidence and remediation specificity. Always treat the model's output as an investigation starting point, not a final verdict; every dependency pair requires human verification before any code change is applied.

This prompt is designed to be called programmatically after a test suite has been executed in both randomized and sequential orders. The implementation harness should collect the raw test execution logs from both runs, normalize timestamps and test names, and pass them into the prompt as structured inputs. Because the output identifies specific dependency pairs, the harness must validate that every returned predecessor-successor pair references tests that actually exist in the input logs. A test name that appears in the output but not in the input is a hallucination and must trigger a retry or human review before the finding is surfaced to a developer.

Wire the prompt into a CI post-execution step or an on-demand investigation tool. The harness should: (1) collect the randomized-run log and sequential-run log as separate text inputs; (2) inject them into the [RANDOMIZED_RUN_LOG] and [SEQUENTIAL_RUN_LOG] placeholders; (3) set [OUTPUT_SCHEMA] to require a JSON array of objects with fields predecessor_test, successor_test, evidence_summary, and confidence; (4) parse the model response and validate that each predecessor_test and successor_test string matches a test name found in the input logs using exact or fuzzy matching; (5) discard or flag any pair where the model's confidence is below a configurable threshold (start with 0.7); and (6) log the full prompt, response, and validation results for auditability. For high-risk codebases such as payments or auth, add a human-in-the-loop gate that requires a platform engineer to confirm each dependency pair before it is filed as a bug.

Model choice matters here. Use a model with strong reasoning capabilities and a context window large enough to hold both full test logs. If logs exceed the context window, preprocess them by extracting only failed test cases and their immediate neighbors from each run, or chunk by test class and run the prompt per class. Do not truncate logs arbitrarily—missing context causes false dependency claims. For retry logic, if validation fails due to hallucinated test names or malformed JSON, retry once with a stricter [CONSTRAINTS] block that explicitly forbids inventing test names and requires line-number citations from the input logs. If the second attempt also fails, escalate to a human with the raw logs and both model responses attached. Avoid wiring this prompt directly into an auto-quarantine action; dependency detection is probabilistic and false positives can waste developer trust.