Schema Evolution Compatibility Test Prompt

This prompt is designed for API platform teams and data engineers who need an automated, deterministic gate in their CI/CD pipeline to assess the safety of a proposed schema change. The core job-to-be-done is to transform a machine-readable schema diff, a semantic versioning policy, and a list of registered API consumers into a structured, actionable test plan. The ideal user is a developer opening a pull request or a release manager preparing a build, who needs to know—before any code reaches staging—whether a change will break existing clients, violate the team's deprecation window, or silently alter behavior for downstream systems. The prompt assumes the input schemas are valid, complete, and formatted in a specification language like OpenAPI, JSON Schema, Protobuf, or Avro.

You should use this prompt when the cost of a breaking change reaching production is high and you want to shift compatibility verification left into the code review process. It is most effective when wired into a pipeline that automatically extracts the 'before' schema from the main branch and the 'after' schema from the feature branch, then feeds both into the prompt alongside a structured policy document. The output is not a set of executable test scripts; it is a detailed test plan that a human or a downstream test harness can implement. This distinction is critical: the prompt is a reasoning and planning tool, not a runtime validator. Do not use it for real-time request/response validation, performance benchmarking, or security authorization logic, as those concerns require specialized tooling and deterministic enforcement.

Before using this prompt, ensure you have a well-defined semantic versioning policy that the model can parse, including rules for major, minor, and patch changes, deprecation timelines, and sunset periods. The prompt's value degrades significantly if the policy is vague or if the consumer registry is incomplete. A common failure mode is providing schemas that are syntactically valid but semantically ambiguous—for example, an OpenAPI spec where a field's type changes from integer to number without an explicit widening annotation. In such cases, the model may flag a false positive or miss a subtle compatibility risk. To mitigate this, always pair this prompt with a deterministic schema linting step that normalizes and validates inputs before they reach the model. If your change involves a regulated domain or a financially significant API surface, the generated test plan must be reviewed by a human before any automated gating decision is made.

This prompt template is designed to be invoked programmatically as part of a schema change review pipeline. It expects a structured diff between two schema versions, along with your organization's compatibility rules and deprecation policies. The model's job is to produce a structured compatibility assessment and a set of executable test scenarios, not to make subjective judgments about design quality. Before using this template, ensure you have a reliable way to extract and format the schema diff—this prompt does not perform diffing itself.

textCopy
You are a schema compatibility analyst. Your task is to assess the backward and forward compatibility of a schema change and generate targeted test scenarios.

## INPUT

[SCHEMA_DIFF]

## CONSTRAINTS

[COMPATIBILITY_RULES]

## DEPRECATION POLICY

[DEPRECATION_POLICY]

## OUTPUT_SCHEMA

Return a single JSON object with the following structure. Do not include any text outside the JSON object.

{
  "assessment": {
    "backward_compatible": boolean,
    "forward_compatible": boolean,
    "breaking_changes": [
      {
        "change": "string (description of the change)",
        "impact": "string (explanation of consumer impact)",
        "affected_consumers": ["string"]
      }
    ],
    "deprecation_violations": [
      {
        "field": "string",
        "violation": "string",
        "policy_reference": "string"
      }
    ]
  },
  "test_scenarios": [
    {
      "id": "string",
      "type": "backward_compatibility | forward_compatibility | deprecation",
      "description": "string",
      "test_input": {},
      "expected_behavior": "string",
      "consumer_version": "string"
    }
  ]
}

## INSTRUCTIONS

1. Analyze [SCHEMA_DIFF] against [COMPATIBILITY_RULES] and [DEPRECATION_POLICY].
2. Identify every breaking change, including additive changes that violate forward compatibility expectations.
3. Flag any deprecation policy violations, citing the specific policy clause.
4. Generate concrete test scenarios that a CI system can execute. Each scenario must include a specific test input and the expected behavior.
5. If no breaking changes or violations are found, return empty arrays for those fields.
6. Do not suggest fixes or design alternatives. Only report findings and generate tests.

To adapt this prompt for your pipeline, replace the three square-bracket placeholders with data from your schema registry and policy store. [SCHEMA_DIFF] should be a structured representation of the change—ideally a JSON or YAML diff, not raw text. [COMPATIBILITY_RULES] should encode your team's specific rules, such as 'adding a required field is backward-incompatible' or 'removing an enum value requires a deprecation period.' [DEPRECATION_POLICY] should include the exact policy text your organization enforces, so the model can cite specific clauses in its violations. If your pipeline already classifies changes before reaching this prompt, you can omit the assessment block and keep only the test scenario generation instructions to reduce token usage and latency.

After copying this template, validate the output against the [OUTPUT_SCHEMA] before accepting it. A malformed JSON response or a missing test_scenarios array should trigger a retry or fallback. For high-risk schema changes—such as those affecting payment, authentication, or data durability APIs—route the generated test scenarios for human review before they are committed to the test suite. Do not rely solely on the model's compatibility assessment; always run the generated tests against actual consumer stubs or recorded traffic to confirm the findings.

This prompt is designed to be called programmatically within a CI/CD pipeline, triggered by a pull request that modifies an API schema file (e.g., OpenAPI, GraphQL, Protobuf). The implementation harness should extract the schema diff, feed it to the prompt, and parse the structured output to block merges or create review tickets. The primary integration point is a GitHub Action, GitLab CI job, or a custom webhook receiver that has access to the base and head schema artifacts.

The harness must construct the [SCHEMA_DIFF] input by running a standard diff tool (like git diff or openapi-diff) against the target and source schema versions. The [DEPRECATION_POLICY] placeholder should be populated from a committed governance file in the repository (e.g., SCHEMA_EVOLUTION.yaml). After the model returns its JSON assessment, a strict validator must check the output against the expected [OUTPUT_SCHEMA]. If compatibility.verdict is BREAKING and the change is not explicitly tagged with a major version bump or an approved exception label, the CI job must fail with a clear message, blocking the merge. Log the full prompt, raw response, and parsed verdict as an artifact for auditability.

For model choice, use a capable long-context model like claude-sonnet-4-20250514 or gpt-4o to handle large diffs. Implement a retry wrapper with up to two attempts if the initial output fails JSON schema validation. If validation fails after retries, the harness should fail closed (block the merge) and escalate for human review. Do not use this prompt for real-time traffic routing decisions; it is a static analysis tool for the change review process. The next step is to integrate the parsed consumer_impact array into your ticketing system, automatically creating follow-up tasks for any identified downstream consumers.