Flaky Test Diagnosis and Repair Prompt

Raw output from the test runner showing the failure, including stack traces, assertion errors, and timestamps.

FAIL: test_user_login (tests/auth/test_session.py:142) AssertionError: Expected 200, got 503 Runtime: 2.3s

Must contain at least one stack trace or assertion error. Null or empty input should abort the prompt and return an error. Validate with regex for common test framework output patterns (pytest, JUnit, Jest).

The full source code of the failing test case, including setup, teardown, and any shared fixtures or helpers it calls.

def test_user_login(db_session): user = create_test_user(db_session) client = TestClient(app) response = client.post('/login', json={'email': user.email, 'password': 'test123'}) assert response.status_code == 200

Must be parseable source code. If the file path is provided instead of inline code, the harness must read the file before calling the model. Validate that the function name matches the failure log.

A unified diff of recent commits to the repository, filtered to files that the failing test depends on or imports.

diff --git a/src/auth/session.py b/src/auth/session.py @@ -45,7 +45,7 @@ def create_session(user_id):

• session = db.Session(user_id=user_id, ttl=3600)

• session = db.Session(user_id=user_id, ttl=300)

Must be a valid unified diff. If empty, the model should note the absence of recent changes as a signal. Validate that the diff applies cleanly to the target branch to avoid hallucinated context.

A structured log of the last N runs of this specific test, including pass/fail status, duration, and the CI environment or machine ID.

Run 1: PASS (2.1s) on ci-worker-3 Run 2: FAIL (2.3s) on ci-worker-3 Run 3: PASS (2.0s) on ci-worker-1 Run 4: FAIL (2.4s) on ci-worker-3

Must include at least 3 runs to establish a pattern. If fewer than 3 runs are available, the model should flag low confidence. Validate that timestamps are sequential and durations are positive numbers.

A mapping of the failing test's dependencies: functions it calls, fixtures it uses, and external services it touches.

{ 'test_user_login': { 'calls': ['create_test_user', 'app.test_client'], 'fixtures': ['db_session'], 'external': ['POST /login', 'database: users table'] } }

Must be a valid JSON object. If generated automatically, validate that all referenced functions exist in the codebase. Null is allowed if the graph cannot be constructed, but the model must note this as a limitation.

The runtime configuration for the environment where the failure occurred, including database versions, feature flags, and resource limits.

Database: PostgreSQL 15.2 Python: 3.11.4 Feature Flags: new_auth_flow=enabled CPU Limit: 2 cores Memory Limit: 4GB

Must be a non-empty string or structured key-value object. Validate that critical fields (database type, language version) are present. If the failure is environment-specific, missing config will degrade diagnosis quality.

A library of previously identified flaky test patterns in this codebase, used to match against the current failure.

[ {'pattern': 'database connection pool exhaustion', 'signature': 'OperationalError: too many connections'}, {'pattern': 'race condition on user creation', 'signature': 'IntegrityError: duplicate key value violates unique constraint'} ]

Must be a valid JSON array. Can be empty if no patterns are cataloged. Validate that each entry has a 'pattern' and 'signature' field. The model should prioritize matching known patterns before generating novel hypotheses.

A summary of the repository structure, testing conventions, and any relevant architectural notes that inform repair suggestions.

This project uses pytest with fixtures defined in conftest.py. Database sessions are scoped to 'function' and use transactional rollback. CI runs tests in parallel with 4 workers. Shared state between tests is discouraged.

Must be a non-empty string. If unavailable, the model should request it or note reduced confidence in repair suggestions. Validate that the context includes testing framework and concurrency model.

string (UUID v4)

Must match UUID v4 regex pattern. Generate a new one if missing.

string

Must match the format of a fully qualified test name from the input log (e.g., 'path/to/test_file.py::TestClassName::test_method_name').

enum string

Must be one of: 'RACE_CONDITION', 'ORDER_DEPENDENCY', 'STATE_LEAK', 'TIMING_ASSUMPTION', 'NETWORK_FLAKE', 'DATA_CONTENTION', 'ENVIRONMENT_DRIFT', 'UNKNOWN'. Reject any other value.

number (0.0 - 1.0)

Must be a float between 0.0 and 1.0 inclusive. If the model outputs a percentage or integer, normalize it. If missing, set to null and flag for human review.

array of strings

Each string must be a verbatim line or block copied from the provided failure log. If the array is empty, the diagnosis must be rejected as ungrounded.

string or null

If root_cause_category is 'STATE_LEAK' or 'ORDER_DEPENDENCY', this field is required and must be a non-empty string. Otherwise, it must be null.

object

Must contain 'file_path' (string) and 'patch_hunk' (string). 'patch_hunk' must be valid unified diff format. If no fix is possible, 'patch_hunk' must be an empty string and a 'rationale' field must be present.

boolean

Must be true if confidence_score < 0.8 or root_cause_category is 'UNKNOWN'. Otherwise, can be false. A false value here does not bypass organizational review policies.

Output identifies a specific root cause category (e.g., race condition, test ordering dependency, network timeout) with supporting evidence from logs or code.

Output provides only a generic symptom description (e.g., 'test failed') or blames the test runner without evidence.

Parse output for a root cause label and verify it is backed by at least one log line, timestamp, or code reference.

The suggested code change eliminates non-deterministic waits (e.g., fixed sleep) and replaces them with deterministic synchronization (e.g., polling, callbacks, or mocks).

The repair introduces a new fixed sleep() or wait() call, or the test still depends on external service availability.

Scan the suggested diff for sleep, wait, or un-mocked network calls. Fail if any are present without a timeout-based escape hatch.

The repair ensures the test does not depend on shared mutable state from other tests. It includes explicit setup or teardown logic to reset the environment.

The test reads from a global variable, static field, or database table that is not reset in a beforeEach or afterEach block.

Check the diff for missing setup/teardown blocks if the root cause involves shared state. Verify no cross-test dependencies remain.

Assertions are relaxed to be order-independent or time-tolerant where appropriate, or tightened to check specific invariants rather than exact system state.

The repair weakens an assertion to assertTrue(true) or removes it entirely without justification, masking the failure rather than fixing it.

Review the assertion diff. Flag any removed assertions or assertions that now trivially pass regardless of system behavior.

The repair includes explicit cleanup of files, database records, or network connections in a finally block or teardown hook.

The test creates resources without corresponding deletion logic, causing subsequent runs to fail due to naming conflicts or resource exhaustion.

Trace resource creation calls in the diff. Confirm a corresponding deletion or cleanup call exists and is executed on both success and failure paths.

The test logic is robust to slow CI environments. It uses dynamic timeouts or event-driven triggers instead of assuming execution speed.

The test passes locally but fails in CI. The repair does not address the CI-specific timing constraint (e.g., a hardcoded 1-second timeout).

Simulate a slow environment by mocking a delayed response. The test should still pass if the repair is correct.

Mocks simulate the specific failure mode identified in the logs (e.g., timeout, 500 error, malformed response) to prevent future regressions.

Mocks are overly permissive or return hardcoded success values that would not catch the original flaky failure condition.

Inject the original failure condition into the mocked dependency. Verify the test fails against the old code and passes with the repair.

The repair adds a diagnostic log statement that would help debug a future recurrence of this flake (e.g., timing delta, resource ID).

The repair removes existing logging or adds no context to help an on-call engineer diagnose a future failure.

Check the diff for a new log statement containing a variable relevant to the root cause. Fail if no diagnostic information is added.