Test Isolation Boundary Audit Prompt

This prompt is designed for platform engineers and QA architects who need to systematically enforce test independence at scale. The core job-to-be-done is converting a suspicion of shared mutable state into a structured, actionable audit report. You should use this prompt when your team is experiencing unexplained flaky tests, test order dependency bugs, or CI failures that disappear on retry. The ideal user has access to the test source code, the relevant production source code, and optionally, test execution logs that can provide runtime evidence of cross-test contamination. The prompt is not a general code review tool; it is a targeted forensic instrument for identifying violations of test isolation boundaries.

To use this prompt effectively, you must provide concrete source code as the [TEST_CODE] and [SOURCE_CODE] inputs. The prompt assumes these are provided as plain text or markdown code blocks. For best results, include the specific test files that are failing and the production modules they exercise. If you have execution logs, provide them in the [EXECUTION_LOGS] placeholder to ground the analysis in runtime behavior, such as unexpected values in static fields or singletons. The prompt is designed to reason about common isolation-breaking patterns: global singletons, static caches, mutable class-level state, shared database records without cleanup, and file system side effects. It will not be effective if the root cause is an infrastructure issue like network timeouts, disk exhaustion, or CI worker resource contention; for those problems, use a dedicated log analysis or environment audit prompt instead.

Before using this prompt, ensure you have ruled out simple environmental failures and that the flaky behavior is correlated with test execution order or parallel execution. The prompt's value is in its ability to trace side effects across test boundaries and produce a ranked violation report with specific refactoring recommendations. Do not use this prompt when you lack access to the source code, as the analysis depends entirely on static code review augmented by log evidence. After running the audit, the output should be reviewed by a senior engineer who can validate the findings and decide on the appropriate refactoring strategy, such as introducing test fixtures with proper setup and teardown, resetting singletons, or moving to dependency injection.

This prompt template is designed to be dropped into your AI workflow when you have a set of test files and their execution context ready for analysis. It instructs the model to act as a platform engineering auditor, systematically scanning for the most common vectors of test pollution: global singletons, static caches, shared file systems, environment variable mutations, and database state leakage. The output is not a generic warning but a structured, actionable report that maps each violation to a specific file and line, explains the cross-test side effect, and proposes a concrete cleanup or refactoring strategy.

markdownCopy
You are a senior platform engineer auditing a test suite for isolation violations. Your goal is to identify any shared mutable state that could cause one test to affect the outcome of another, leading to flaky or order-dependent failures.

Analyze the provided test code and any supplied execution logs. For each violation found, you must produce a structured report entry with the following fields:
- `violation_id`: A unique identifier for the finding (e.g., `ISO-001`).
- `severity`: One of `CRITICAL`, `HIGH`, `MEDIUM`, or `LOW`.
- `category`: The type of violation (`GLOBAL_SINGLETON`, `STATIC_CACHE`, `SHARED_FILESYSTEM`, `ENV_VAR_MUTATION`, `DATABASE_STATE_LEAK`, `OTHER`).
- `source_location`: The exact file path and line number where the shared state is defined or mutated.
- `affected_tests`: A list of test names or functions that read or write this shared state.
- `side_effect_description`: A clear, one-sentence explanation of how this state causes cross-test interference.
- `recommended_fix`: A specific, code-level suggestion for isolating the state (e.g., use a fixture with teardown, reset a singleton in `beforeEach`, use unique temp directories).

[TEST_CODE]
[EXECUTION_LOGS]

Before the report, include a brief summary stating the total number of violations found and the most common category. If no violations are found, state that explicitly and do not fabricate issues.

Output the final report as a JSON object with a `summary` string and a `violations` array.

To adapt this prompt, replace the [TEST_CODE] placeholder with the full source of your test files, ensuring you include any imported modules or utility functions that might contain global state. The [EXECUTION_LOGS] placeholder should be filled with the output from a test run that exhibited flaky or order-dependent failures; logs from a randomized test run (e.g., using pytest-randomly or a similar tool) are particularly valuable. For high-risk CI pipelines, always route the model's output through a validation step that checks the JSON schema and verifies that every source_location points to a real line in the provided code. A human platform engineer should review all CRITICAL and HIGH severity findings before any automated refactoring is applied.

The Test Isolation Boundary Audit Prompt is designed to run as a pre-merge or nightly CI gate, not as a real-time lint step. Because it requires access to the full test suite source code, execution logs, and optionally runtime state snapshots, it should be invoked after test execution completes and artifacts are available. The prompt expects [TEST_SUITE_CODE] (the test files themselves), [EXECUTION_LOGS] (sequential and randomized run output), and [RUNTIME_STATE_SNAPSHOTS] (optional database dumps, cache contents, or singleton state captures). Wire this into your CI platform (GitHub Actions, GitLab CI, Jenkins) as a post-test job that collects these artifacts, assembles the prompt, and calls your model API.

Validation and retry logic is critical because the output is a structured isolation violation report. Implement a JSON schema validator that checks for the required fields: violation_id, shared_state_type (one of global_singleton, static_cache, database_row, file_system, environment_variable, in_memory_collection), affected_tests (array of test names), evidence (log excerpts or state diffs), severity (one of critical, high, medium, low), and remediation. If the model output fails schema validation, retry once with the validation errors appended to [CONSTRAINTS]. If the retry also fails, log the raw output and escalate to a platform engineer via a Slack webhook or ticketing system. Do not silently accept malformed reports—an unparseable audit is worse than no audit because it creates a false sense of security.

Model choice and context window matter here. Test suites can be large, and execution logs even larger. Use a model with at least a 128K context window (Claude 3.5 Sonnet, GPT-4 Turbo, or Gemini 1.5 Pro) and consider chunking large test suites by module or directory. If chunking, run the prompt independently per chunk and then run a second cross-chunk reconciliation pass that looks for shared state violations spanning module boundaries (e.g., a global singleton mutated in one module and read in another). Store each audit report as a versioned artifact in your CI run so you can track isolation health over time and detect regressions when a previously clean module starts showing violations.

Human review is required before auto-applying remediations. The prompt produces refactoring recommendations, but these can be destructive—suggesting a test order dependency fix might mask a real bug, and recommending singleton teardown might break production code that relies on that state. Configure the harness to post the audit report as a comment on the pull request (if pre-merge) or as a ticket in your issue tracker (if nightly). Require a platform engineer or test owner to acknowledge each critical or high severity finding before the remediation is applied. For medium and low findings, you can auto-create backlog tickets but should not auto-apply fixes. Log every audit run, validation result, retry, and human decision for auditability and to measure the prompt's precision and recall over time.