Dependency Update CI Failure Triage Prompt

This prompt is designed for the moment a CI pipeline fails on a branch where the primary and intentional change is a dependency version bump. The core job-to-be-done is rapid classification: you need to determine if the failure is a direct consequence of the update, a pre-existing flaky test that happened to trigger, or an environment drift issue unrelated to the code change. The ideal user is a developer or platform engineer staring at a red build who needs to decide within minutes whether to revert the dependency, push a fix, or suppress a known flaky test. It assumes you are not performing a full security audit or a comprehensive incident postmortem, but rather a focused triage to unblock the pipeline.

To use this prompt effectively, you must provide three concrete inputs: the full build log, the isolated test failure output (stack traces, assertion errors, and test names), and the dependency diff showing exactly which packages changed and how. Without all three, the model cannot reliably distinguish between a breaking API change in the dependency and a test that has been failing intermittently for weeks. The prompt is most effective when the dependency change is a single package or a small, related group; for a large batch update touching dozens of packages, you should first bisect the changes or use the 'Dependency Update Risk Assessment Prompt' to narrow the field before triaging individual failures.

Do not use this prompt when the failure involves a security advisory, a license violation, or a suspected malicious package. Those scenarios require the 'Security Advisory Impact Analysis Prompt', 'Dependency License Compliance Review Prompt', or 'Malicious Package Detection Heuristic Prompt' respectively, as they demand specialized output schemas and evidence handling. Similarly, if the failure occurs in production after a deployment, this prompt is insufficient; you should use the 'Dependency Update Rollback Decision Prompt' which synthesizes production metrics and user impact signals. After classification, if the root cause is confirmed as update-induced, your next step is to pair this output with the 'Breaking Change Detection Prompt for Package Manifests' to map the failure to specific API changes and affected call sites in your codebase.

This template is the core instruction set you will send to the model. It is designed to receive structured and unstructured CI failure data and return a root cause classification with supporting evidence. The prompt uses square-bracket placeholders for all dynamic inputs, making it safe to paste directly into your AI harness, a Python script, or a CI/CD plugin. Before using it, ensure you have collected the build log, the dependency diff, and any relevant test failure outputs. The prompt is opinionated about output structure to ensure the result can be parsed by downstream automation, but it also includes a [RISK_LEVEL] parameter to adjust the model's caution when suggesting fixes.

textCopy
You are an on-call reliability engineer investigating a CI pipeline failure that occurred immediately after a dependency update.

Your task is to analyze the provided build log, test output, and dependency change information to determine the root cause of the failure. You must classify the failure into one of three categories: `CAUSED_BY_UPDATE`, `PRE_EXISTING_FLAKE`, or `ENVIRONMENT_DRIFT`. You must then provide a structured analysis.

# INPUTS

[DEPENDENCY_DIFF]
[BUILD_LOG]
[TEST_OUTPUT]
[REPOSITORY_CONTEXT]

# OUTPUT_SCHEMA

You must respond with a single JSON object conforming to this structure:
{
  "classification": "CAUSED_BY_UPDATE" | "PRE_EXISTING_FLAKE" | "ENVIRONMENT_DRIFT",
  "confidence": 0.0-1.0,
  "primary_evidence": [
    {
      "source": "build_log" | "test_output" | "dependency_diff",
      "snippet": "string",
      "explanation": "string"
    }
  ],
  "alternative_hypotheses": [
    {
      "classification": "string",
      "plausibility": "string",
      "counter_evidence": "string"
    }
  ],
  "fix_direction": "string or null",
  "requires_human_judgment": true | false
}

# CONSTRAINTS

- Base your classification on evidence from the logs and diffs, not on assumptions.
- If the failure is a `PRE_EXISTING_FLAKE`, you must cite evidence from the build log that shows the same test failing on a previous, unrelated build.
- If the failure is `ENVIRONMENT_DRIFT`, you must identify the specific environmental factor (e.g., missing system package, Node.js version mismatch, network timeout) that is not a direct result of the declared dependency change.
- For a `CAUSED_BY_UPDATE` classification, your `fix_direction` must reference a specific breaking change from the dependency diff or a new transitive conflict.
- If confidence is below 0.7, `requires_human_judgment` must be set to `true`.
- Do not hallucinate file paths or function names that are not present in the provided inputs.

# RISK_LEVEL: [RISK_LEVEL]

- If RISK_LEVEL is "high", you must be more conservative in your `fix_direction` and explicitly recommend a rollback of the dependency as the immediate next step.
- If RISK_LEVEL is "low", you may suggest a code fix or configuration change as the `fix_direction`.

To adapt this template, replace the square-bracket placeholders with your actual data. [DEPENDENCY_DIFF] should contain the output of a command like git diff HEAD~1 -- package.json package-lock.json or a similar manifest diff. [BUILD_LOG] and [TEST_OUTPUT] should be the raw text from your CI system. [REPOSITORY_CONTEXT] can include a link to the repository, a brief description of the project's purpose, or the runtime environment. The [RISK_LEVEL] placeholder should be set to either "high" or "low" based on whether the pipeline is for a production deployment or a development branch. After running the prompt, validate the output JSON against the schema before allowing any automated rollback or ticket creation to proceed.

This prompt is designed to sit inside a CI failure handler—either as a post-build step in your pipeline (GitHub Actions, GitLab CI, Jenkins) or as a CLI tool that developers invoke locally with a failed log file. The primary integration point is after dependency update steps (e.g., npm update, pip install --upgrade, cargo update, Dependabot, Renovate) and before a human starts debugging. The prompt expects structured inputs: the dependency change list, the full CI failure log, and any relevant environment context. Wire it so that the handler extracts these from the CI environment automatically—parse the lock file diff, capture stdout/stderr from the failed step, and include environment metadata (OS, Node/Python/Rust version, CI runner type) as [CONTEXT].

Validation and output contract: The prompt returns a JSON object with root_cause_classification (one of dependency_update, pre_existing_flaky_test, environment_drift, inconclusive), confidence (0.0–1.0), evidence (array of log line references), and fix_direction (string). Validate this output before surfacing it to the developer. If confidence is below 0.6 or root_cause_classification is inconclusive, route to a human triage queue rather than auto-suggesting a fix. If the JSON schema is malformed, retry once with a repair prompt that includes the raw output and the expected schema. Model choice: Use a model with strong reasoning on structured logs—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models for this task; log triage requires holding long context and correlating distant failure signals. Tool use: If your harness supports it, give the model access to a git diff tool to inspect the exact dependency changes and a test_history tool to check whether failing tests have flaked in recent runs. This grounds the classification in evidence beyond the log text alone.

Logging and observability: Log every triage attempt with the prompt version, model, input hashes, output classification, confidence, and whether the developer accepted or overrode the result. This creates a feedback loop for prompt improvement. Track the distribution of classifications over time—if environment_drift spikes, your CI runners may have a systemic issue. What to avoid: Do not auto-apply fix directions without human review. A fix_direction like 'pin dependency to previous version' or 'update test assertion' is a suggestion, not a command. The prompt reduces mean time to diagnosis, not mean time to resolution. Always keep the human in the loop for the fix step, especially when the confidence score is moderate. For high-risk production pipelines, add an approval gate: the triage output is posted as a PR comment or Slack thread, and a developer must acknowledge it before the pipeline proceeds.