Third-Party API Flakiness Attribution Prompt

This prompt is designed for integration test engineers who are investigating a flaky test and have already narrowed the failure domain to an interaction with an external, third-party API. The core job-to-be-done is to move beyond a hunch and produce a defensible, structured attribution decision. You should use this prompt when you have collected structured evidence from both passing and failing test runs, including HTTP response payloads, status codes, latency distributions, and rate-limiting headers. The prompt acts as a rigorous analyst, cross-referencing this evidence to determine if the root cause is a dependency issue (e.g., a provider outage, a rate limit, a slow backend) or an application-level problem (e.g., a brittle assertion, a missing retry, a timeout misconfiguration).

This prompt is not a replacement for checking a provider's status page or running network diagnostics. Instead, it structures the evidence you already possess into a clear, repeatable analysis. For example, you might feed it a set of logs showing a 502 Bad Gateway from the provider alongside a spike in latency, and the prompt will guide the model to attribute the failure to the provider rather than your code. Conversely, if the evidence shows consistent 200 OK responses with a valid payload but your test fails on a strict schema assertion, the prompt will correctly attribute the flakiness to your application's validation logic. The ideal user is someone who needs to make a quick, evidence-backed decision on whether to mock the dependency, adjust timeouts and retries, or escalate the issue to the external provider's support team.

Do not use this prompt for initial flakiness discovery or for analyzing failures that are clearly unrelated to external API calls, such as unit test logic errors or database deadlocks. It is also unsuitable when you lack concrete API interaction logs from both passing and failing scenarios, as the analysis would be speculative. The output is an attribution, not a full root cause fix. After receiving the attribution, your next step should be to act on the recommendation: implement a mock for provider-side issues, adjust your application's resilience patterns for client-side timeout issues, or open a ticket with the provider if their SLA has been breached. Avoid using this prompt as a final sign-off without a human reviewing the evidence for high-severity or customer-impacting test failures.

This section provides the core prompt template for the Third-Party API Flakiness Attribution workflow. The prompt is designed to receive structured evidence from your test harness—including API response metadata, error codes, latency distributions, and rate-limiting signals—and produce an evidence-based attribution report. The template uses square-bracket placeholders that you must replace with real data from your test runs before sending the prompt to the model. Each placeholder corresponds to a specific input your integration test infrastructure should collect and format.

textCopy
You are an integration test analyst specializing in third-party API dependency failures.
Your task is to determine whether a flaky test failure is caused by the external API, the application code, or the test environment.

## INPUT DATA

### Test Context
- Test name: [TEST_NAME]
- Test suite: [TEST_SUITE]
- Execution timestamp: [EXECUTION_TIMESTAMP]
- CI run URL: [CI_RUN_URL]
- Application version: [APP_VERSION]

### API Call Details
- Endpoint: [API_ENDPOINT]
- HTTP method: [HTTP_METHOD]
- Request payload (sanitized): [REQUEST_PAYLOAD]
- Response status code: [RESPONSE_STATUS_CODE]
- Response body (sanitized, first 2000 chars): [RESPONSE_BODY]
- Response headers (relevant): [RESPONSE_HEADERS]
- Request duration (ms): [REQUEST_DURATION_MS]
- Timeout configured (ms): [TIMEOUT_CONFIGURED_MS]
- Retry count before failure: [RETRY_COUNT]

### Latency Distribution (last 10 calls to same endpoint)
[LATENCY_DISTRIBUTION_TABLE]

### Rate Limiting Signals
- Rate limit headers present: [RATE_LIMIT_HEADERS]
- 429 responses in last 5 minutes: [RECENT_429_COUNT]
- Retry-After header value: [RETRY_AFTER_VALUE]

### Application Error Context
- Application error message: [APP_ERROR_MESSAGE]
- Stack trace (first 1000 chars): [STACK_TRACE]
- Error handling code path: [ERROR_HANDLING_PATH]

### Environment Context
- Network zone: [NETWORK_ZONE]
- DNS resolution time (ms): [DNS_RESOLUTION_MS]
- TLS handshake time (ms): [TLS_HANDSHAKE_MS]
- Proxy or gateway in path: [PROXY_DETAILS]

## OUTPUT SCHEMA

Produce a JSON object with this exact structure:
{
  "attribution": {
    "primary_category": "THIRD_PARTY_API" | "APPLICATION_CODE" | "TEST_ENVIRONMENT" | "NETWORK_INFRASTRUCTURE" | "INCONCLUSIVE",
    "confidence": 0.0-1.0,
    "secondary_categories": ["string"]
  },
  "evidence": [
    {
      "signal": "string (specific observation)",
      "strength": "STRONG" | "MODERATE" | "WEAK",
      "supports_category": "string",
      "explanation": "string (why this signal points to the category)"
    }
  ],
  "counter_evidence": [
    {
      "signal": "string (observation that contradicts primary attribution)",
      "strength": "STRONG" | "MODERATE" | "WEAK",
      "explanation": "string"
    }
  ],
  "recommendations": {
    "immediate_action": "string (what to do right now for this test)",
    "mock_or_stub_strategy": "string (how to simulate this API in tests)",
    "code_changes": ["string (specific application code changes if applicable)"],
    "monitoring_improvements": ["string (observability gaps to close)"]
  },
  "requires_human_review": true | false,
  "human_review_reason": "string (if requires_human_review is true)"
}

## CONSTRAINTS
- Do not attribute to THIRD_PARTY_API unless you have specific evidence from response codes, headers, latency patterns, or rate limiting signals.
- Do not attribute to APPLICATION_CODE unless you see error handling gaps, missing retries, or incorrect timeout configuration in the application context.
- If latency is high but the API eventually returns a valid response, consider whether the application timeout is too aggressive.
- If the API returns 5xx errors, check whether the application has retry logic and whether retries were exhausted.
- If rate limiting signals are present, attribute to THIRD_PARTY_API regardless of other factors.
- If evidence is contradictory or insufficient, set primary_category to INCONCLUSIVE and confidence below 0.5.
- Set requires_human_review to true for any attribution with confidence below 0.7 or when primary_category is INCONCLUSIVE.
- Never recommend production code changes without flagging for human review.

## EXAMPLES

### Example 1: Clear Third-Party Failure
Input: 503 status, no retry-after header, latency distribution shows all recent calls failing with 503, no application error handler gap.
Output attribution: THIRD_PARTY_API, confidence 0.95.

### Example 2: Application Timeout Misconfiguration
Input: 200 status but response time 4500ms, timeout configured at 3000ms, latency distribution shows consistent 4000-5000ms responses.
Output attribution: APPLICATION_CODE, confidence 0.85, recommendation: increase timeout or add async pattern.

### Example 3: Rate Limiting
Input: 429 status, Retry-After: 60, 12 rate-limited responses in last 5 minutes.
Output attribution: THIRD_PARTY_API, confidence 0.98, recommendation: implement exponential backoff with jitter.

To adapt this template for your environment, replace each square-bracket placeholder with data collected from your test execution framework. The latency distribution table should be formatted as a markdown table with columns for timestamp, duration, status code, and any error codes. If a placeholder has no data available, replace it with N/A rather than leaving it empty—this prevents the model from hallucinating missing information. The output schema is designed to be machine-parseable so your test harness can automatically route attributions to the right team: third-party API issues go to the API integration owner, application code issues go to the development team, and environment issues go to platform engineering. Always run the output through a JSON schema validator before acting on the recommendations, and ensure human review is triggered when the requires_human_review field is true.

This prompt is designed to sit inside an automated flakiness triage pipeline, not as a one-off chat interaction. The typical integration point is a CI/CD post-execution hook or a dedicated flakiness analysis service that receives test failure artifacts (logs, HAR files, API response traces, timing data) and routes them to the LLM for attribution. The harness should collect all required inputs before calling the model: the failing test name, the full API interaction log including request/response pairs with headers and status codes, latency distributions across multiple runs, any rate-limit headers (X-RateLimit-Remaining, Retry-After), and the application's retry/timeout configuration. Missing or incomplete inputs are the most common cause of incorrect attributions, so the harness must validate input completeness before invoking the prompt.

Wire the prompt into an application function that constructs the [API_INTERACTION_LOG] placeholder from structured trace data rather than raw text dumps. Parse your observability exports (OpenTelemetry spans, HAR files, or API gateway logs) into a normalized JSON structure with fields for timestamp, method, url, request_headers, request_body, status_code, response_headers, response_body (truncated), and latency_ms. Feed this structured log into the prompt template along with the [TEST_NAME], [RETRY_CONFIG], and [ENVIRONMENT_CONTEXT] placeholders. After the model returns its attribution JSON, run a schema validator to confirm the output contains the required fields: attribution_target (one of third_party, application_code, network_infrastructure, unknown), confidence_score (0.0-1.0), evidence_summary, error_pattern_category, and mock_stub_recommendation. Reject and retry outputs that fail schema validation or produce confidence scores below your configured threshold (typically 0.7 for automated routing, lower for human-review queues).

For high-risk pipelines—such as release-blocking test suites or production canary analysis—route attributions with confidence_score < 0.85 or attribution_target == 'unknown' to a human review queue. Log every attribution decision with the full prompt input, model output, validator results, and reviewer action for auditability. Choose a model with strong structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent) and set temperature=0 to minimize variance across repeated runs. Avoid using this prompt for real-time request path decisions; it is an offline analysis tool. The most common production failure mode is stale or truncated API logs that omit the rate-limit headers or response bodies needed for accurate attribution—build log completeness checks into your harness before calling the model.