Dependency Update Test Gap Analysis Prompt

This prompt is designed for QA engineers and developers who need to systematically identify testing gaps introduced by a dependency update. The core job-to-be-done is to prevent production regressions by analyzing a changelog or release diff against the actual usage of that dependency in your codebase. The prompt produces a prioritized list of new or modified test cases that should be written before the update is merged, focusing on behavioral changes, deprecations, and new API surfaces that existing tests do not cover.

To use this prompt effectively, you must provide three concrete inputs: the full changelog or release notes for the dependency update, a representative sample of the codebase showing how the dependency is currently imported and called, and a summary of the existing test coverage for those call sites. The prompt is most effective for minor and major version updates where behavioral changes are documented. It is not a substitute for running your existing test suite, nor is it designed for zero-day vulnerability analysis or license compliance checks—those require separate, specialized prompts. Do not use this prompt when the changelog is empty, the update is a patch version with no documented changes, or the codebase usage is so abstracted that call-site analysis is impossible.

After running this prompt, treat the output as a test planning artifact, not a final test suite. Each recommended test case should be reviewed by a developer familiar with the affected module. For high-risk updates—such as database drivers, authentication libraries, or serialization frameworks—require human sign-off on the test plan before implementation. Wire the output into your issue tracker as a checklist, and consider running this prompt as a mandatory step in your dependency update pull request template.

The template below is designed to be copied directly into your AI harness, prompt management system, or orchestration layer. It uses square-bracket placeholders for all dynamic inputs, ensuring you can swap in changelogs, codebase usage maps, and output format requirements without modifying the core instruction structure. The prompt instructs the model to reason about the delta between what changed in the dependency and what your test suite actually exercises, then produce a prioritized list of missing test cases.

textCopy
You are a QA automation engineer analyzing a dependency update for test coverage gaps.

## INPUT
[DEPENDENCY_NAME]: [PACKAGE_NAME]
[CURRENT_VERSION]: [VERSION_STRING]
[TARGET_VERSION]: [VERSION_STRING]
[CHANGELOG]:

[FULL_CHANGELOG_TEXT]

codeCopy
[CODEBASE_USAGE_MAP]:

[A structured mapping of which files import or call which functions/classes from the dependency. Include call sites, configuration usage, and any wrapper abstractions.]

codeCopy
[EXISTING_TEST_PLAN]:

[A summary of the existing test suite relevant to this dependency: test names, what they cover, and any known coverage gaps.]

codeCopy

## CONSTRAINTS
- Only flag behavior changes that are actually reachable from our codebase based on [CODEBASE_USAGE_MAP].
- Ignore changelog entries for features, APIs, or configuration options we do not use.
- For each identified gap, state whether it is a regression risk, a compatibility risk, or a behavioral change risk.
- Do not recommend tests for performance-only changes unless they alter documented behavior.
- If the changelog contains security fixes, flag them separately with a [SECURITY] tag.

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
{
  "analysis_summary": "A 2-3 sentence summary of the update's risk profile given our usage.",
  "test_gaps": [
    {
      "id": "TG-001",
      "risk_type": "regression | compatibility | behavioral_change | security",
      "changelog_entry": "The exact changelog line or paragraph that triggered this gap.",
      "affected_usage": "Which code paths or call sites are affected, from CODEBASE_USAGE_MAP.",
      "existing_coverage": "What existing tests cover this area, if any, and why they are insufficient.",
      "recommended_test": "A concrete, actionable test case description including inputs, expected behavior, and assertions.",
      "priority": "critical | high | medium | low",
      "effort_estimate": "small | medium | large"
    }
  ],
  "safe_updates": [
    {
      "changelog_entry": "Changelog entries that are safe given our usage.",
      "reason": "Why this change does not require new tests."
    }
  ],
  "security_advisories": [
    {
      "cve_id": "CVE-XXXX-XXXXX if present, else 'N/A'",
      "description": "Brief description of the security fix.",
      "reachable": true | false,
      "recommended_action": "What to do if reachable."
    }
  ]
}

## INSTRUCTIONS
1. Parse [CHANGELOG] and extract every behavior change, deprecation, breaking change, and security fix.
2. Cross-reference each change against [CODEBASE_USAGE_MAP] to determine reachability.
3. For each reachable change, compare against [EXISTING_TEST_PLAN] to identify coverage gaps.
4. Prioritize gaps: critical for breaking changes and security fixes we use, high for behavioral changes in critical paths, medium for edge cases, low for unlikely scenarios.
5. If a changelog entry is ambiguous about whether behavior changed, flag it as a gap with a note about ambiguity.
6. Output only the JSON object. No markdown fences, no commentary outside the JSON.

To adapt this template for your pipeline, replace each bracketed placeholder with data sourced from your package manager, version control, and test management systems. The [CODEBASE_USAGE_MAP] is the most critical input—without an accurate map of which functions and classes your code actually calls, the model will either over-flag irrelevant changes or miss reachable ones. Generate this map with a static analysis tool or a grep-based script before invoking the prompt. The [EXISTING_TEST_PLAN] should be a structured summary, not raw test files; consider using a separate summarization step if your test suite is large. For high-risk updates, always route the output through a human reviewer before merging, and validate the JSON schema programmatically to catch malformed responses before they enter your issue tracker.

This prompt is designed to be called programmatically as part of a pull request check or a scheduled QA automation job. The primary integration point is a CI pipeline step that triggers after a dependency update PR is opened. The harness should fetch the changelog for the updated package, extract the relevant codebase usage via grep or a static analysis tool, and assemble the [CHANGELOG] and [CODE_USAGE] inputs before calling the model. The output is a structured JSON list of test gaps, each with a priority, rationale, and a recommended test case description. This structured output is then consumed by downstream systems: a test management tool to create ticket drafts, a QA dashboard for review, or a notification system to alert the responsible team.

The implementation must include a validation layer that parses the model's JSON output and checks for schema compliance. Each recommended test case object must have non-null priority, untested_behavior, and recommended_test fields. If validation fails, the harness should retry the prompt once with a repair instruction that includes the specific validation error. Logging is critical: capture the full prompt, the raw model response, the parsed output, and the validation result for every run. This trace data is essential for debugging false positives and for tuning the prompt over time. For model choice, a capable mid-tier model like claude-3.5-sonnet or gpt-4o is appropriate; the task requires strong reasoning over code and changelog semantics but does not need the latency of a small model or the cost of a frontier model for every run. The prompt's [RISK_LEVEL] parameter should be set based on the dependency type: set to high for core frameworks and security libraries, medium for utility libraries, and low for dev-only tooling. This parameter gates the inclusion of a human review step in the workflow.

The final stage of the harness is a human review gate for high-risk findings. When the model returns test gaps with a priority of critical or high, the workflow should not automatically create test tickets. Instead, it should post the findings as a comment on the dependency update PR and request a QA lead or senior developer to approve, reject, or modify each recommendation before any tickets are filed. This prevents the automation from flooding the backlog with low-value tests while ensuring that genuinely dangerous untested paths receive immediate attention. Avoid wiring this prompt directly into an auto-merge gate without human review; a false negative where a real test gap is missed could allow a breaking change to reach production.