Mock and Stub Misconfiguration Diagnosis Prompt Template

This prompt is for developers and test infrastructure engineers who are debugging tests that pass in one environment but fail in another—typically passing locally while failing in CI, or vice versa. The core job-to-be-done is comparing mock configurations, stub behaviors, and expected-versus-actual interaction patterns across environments to identify the root cause of the discrepancy. The prompt assumes you have already ruled out genuine application logic bugs, race conditions, and infrastructure failures, and that the most likely culprit is a misconfiguration in how mocks, stubs, or test doubles are set up or behaving differently between environments.

To use this prompt effectively, you must provide concrete evidence from both environments: the test code itself, mock or stub setup files (e.g., jest.mock() calls, sinon.stub() configurations, unittest.mock patches, or WireMock mappings), CI execution logs showing the failure, and local execution output showing the passing case. The prompt works by systematically comparing the mock configurations, expected interaction contracts (expected calls, arguments, return values), and actual interaction patterns (actual calls received, unexpected interactions) across the two environments. It produces a structured discrepancy report with fix suggestions grounded in the provided evidence, not speculation. The prompt is most effective when the mock framework and language are consistent across environments—if you're comparing a Python unittest.mock setup against a Java Mockito configuration, the prompt will need additional translation context.

Do not use this prompt when the failure is caused by genuine timing issues, actual service unavailability in CI, environment variable differences unrelated to mocking, or application code changes that introduced real bugs. This prompt is also not suitable for diagnosing flaky tests that pass and fail intermittently in the same environment—those require a different approach focused on race conditions, test ordering dependencies, or resource contention. Before running this prompt, verify that the test failure is consistently reproducible in one environment and consistently passing in another, and that you have access to the full mock setup code and execution logs from both sides. If you cannot provide evidence from both environments, the prompt will produce low-confidence results that may waste investigation time.

This section provides a complete, copy-ready prompt template for diagnosing mock and stub misconfiguration issues in your test suite. The template is designed to be pasted directly into your AI coding agent or LLM interface. It guides the model through a structured comparison of mock configurations, stub behaviors, and expected-versus-actual interaction patterns to identify the root cause of discrepancies—particularly the common scenario where tests pass locally but fail in CI, or vice versa. Replace every square-bracket placeholder with your actual test code, failure logs, mock setup files, and environment configuration before running the prompt.

markdownCopy
## Role
You are a senior test infrastructure engineer specializing in mock and stub behavior analysis. Your task is to diagnose why a test that uses mocks or stubs is failing, with particular attention to discrepancies between local and CI execution environments.

## Inputs
- **Failing Test Code:** [PASTE_FAILING_TEST_CODE_HERE]
- **Mock/Stub Configuration:** [PASTE_MOCK_SETUP_CODE_HERE]
- **Test Failure Output:** [PASTE_FULL_FAILURE_LOG_OR_STACK_TRACE_HERE]
- **Local Environment Details:** [DESCRIBE_LOCAL_ENV_OS_DEPS_VERSIONS]
- **CI Environment Details:** [DESCRIBE_CI_ENV_OS_DEPS_VERSIONS]
- **Expected Behavior Description:** [DESCRIBE_WHAT_THE_TEST_SHOULD_VERIFY]
- **Passing Reference (optional):** [PASTE_PASSING_TEST_OR_LOG_IF_AVAILABLE]

## Task
1. Parse the failing test to identify every mock/stub dependency and its expected behavior.
2. Compare the mock configuration against the actual test execution to detect mismatches in:
   - **Return values:** Expected vs. actual return types, shapes, or values.
   - **Call expectations:** Expected call count, argument matchers, call order.
   - **Side effects:** Configured side effects that may not execute as expected.
   - **Lifecycle:** Setup/teardown ordering that may cause state leakage.
   - **Environment differences:** OS, filesystem, network, or timing factors that differ between local and CI.
3. For each mismatch found, classify its severity: **CRITICAL** (causes the failure), **CONTRIBUTING** (worsens flakiness), or **BENIGN** (present but unrelated).
4. Identify whether the root cause is in the **test code**, the **mock configuration**, the **production code under test**, or the **environment**.

## Output Schema
Return a JSON object with this exact structure:
```json
{
  "summary": "One-sentence root cause summary.",
  "root_cause_category": "TEST_CODE | MOCK_CONFIG | PRODUCTION_CODE | ENVIRONMENT",
  "findings": [
    {
      "severity": "CRITICAL | CONTRIBUTING | BENIGN",
      "category": "RETURN_VALUE | CALL_EXPECTATION | SIDE_EFFECT | LIFECYCLE | ENVIRONMENT",
      "location": "File:line or mock identifier",
      "expected": "What the test or mock expected.",
      "actual": "What actually occurred.",
      "fix_suggestion": "Concrete, actionable fix."
    }
  ],
  "local_vs_ci_differential": "Explanation if environment difference is the cause, otherwise null.",
  "recommended_fix_order": ["Ordered list of fix steps, most critical first."]
}

Constraints

• Do not guess. If evidence is insufficient for a finding, mark it as INCONCLUSIVE with a note about what additional information is needed.
• Cite specific line numbers or mock identifiers in the location field.
• If the test passes locally but fails in CI, prioritize environment differential analysis.
• If the test fails in both environments, prioritize mock configuration and test logic analysis.

After pasting the template, adapt it by filling in the placeholders with real data from your debugging session. The most critical inputs are the failing test code, the mock/stub configuration, and the full failure output—without these, the model cannot perform a meaningful comparison. If you have a passing reference (e.g., a similar test that works, or a previous passing run log), include it to help the model calibrate its expectations. For environment-specific failures, be precise about OS, dependency versions, and any containerization differences between local and CI. The output schema is designed to be machine-readable so you can pipe the findings directly into your issue tracker or test repair workflow. Always review the recommended_fix_order critically before applying changes—the model may suggest fixes that require human judgment about test intent and architectural trade-offs.

This prompt is designed to be called as part of an automated CI failure investigation pipeline or a developer CLI tool. The primary integration point is immediately after a test suite reports a failure where the discrepancy between local and CI execution suggests a mock, stub, or environment configuration issue. The application harness should collect the failing test code, the mock/stub configuration files, any relevant environment variables or CI configuration, and the full failure output before assembling the prompt.

Input assembly and validation: Before calling the model, validate that all required placeholders are populated with non-empty strings. The [FAILING_TEST_CODE] must contain the actual test function or method, not just the test name. The [MOCK_CONFIGURATION] should include the mock setup code (e.g., jest.mock, sinon.stub, unittest.mock.patch) and any fixture or factory definitions the mocks depend on. The [CI_ENVIRONMENT_VARIABLES] and [LOCAL_ENVIRONMENT_VARIABLES] should be sanitized to remove secrets before inclusion. If the test failure output exceeds 8,000 tokens, truncate it to the last 8,000 tokens and prepend a note: [Failure output truncated to last 8,000 tokens. Full output available in CI logs.]. Model choice: Use a model with strong code reasoning capabilities (e.g., Claude 3.5 Sonnet, GPT-4o) rather than a smaller or faster model, because diagnosing mock misconfigurations requires precise reasoning about test doubles, module resolution, and execution order. Set temperature to 0.0 or 0.1 to maximize deterministic, evidence-grounded output.

Output parsing and validation: Parse the model's JSON response and validate the discrepancies array. Each discrepancy object must have a non-empty evidence field that references a specific line, configuration value, or behavioral difference from the input. If any discrepancy has an empty evidence field or the root_cause_category is not one of the expected enum values (mock_return_value, stub_not_called, module_resolution, environment_difference, timing_dependency, unmocked_dependency), reject the output and retry once with an additional constraint appended: [CONSTRAINT: Every discrepancy must cite specific evidence from the provided inputs. Do not speculate without grounding.]. Retry and escalation: If the second attempt also fails validation, log the raw output and the validation errors, then escalate to a human reviewer with a pre-formatted summary of what the model produced and which validation rules failed. Do not automatically retry beyond two attempts. Logging and traceability: Log the full prompt, model response, validation result, and any retry attempts to your prompt observability system. Tag each log entry with the CI build ID and test name so that on-call engineers can correlate the diagnosis attempt with the original failure. Human review gate: For any diagnosis that recommends changing production configuration or shared test fixtures, flag the output for human review before applying the fix. The fix_suggestions array should be treated as advisory, not auto-applied, unless the change is scoped to a single test file and the suggestion includes a specific code diff that passes your existing pre-commit hooks.