Code Output Diff Against Expected Output Prompt Template

This prompt is designed for evaluation engineers and MLOps teams who need to automatically compare AI-generated code against a known-correct reference implementation. The primary job-to-be-done is gating a code-generation model before deployment by producing a structured diff analysis that categorizes differences as cosmetic, semantic, or functional, and integrates test-case pass/fail results. You should use this when you have a golden dataset of expected code outputs and need to determine whether a model change, prompt update, or new deployment is safe to ship. The ideal user has a reference implementation in hand, a set of test cases that can be executed against the generated code, and a need for automated, repeatable evaluation rather than manual spot-checking.

This prompt is not appropriate for general code review of human-written pull requests, for evaluating code style without a reference implementation, or for assessing code quality in the absence of executable test cases. It assumes you have a ground-truth reference—if you are evaluating code where multiple valid implementations exist, consider using a semantic similarity or rubric-based approach instead. The prompt requires several concrete inputs to function: a reference code block, the AI-generated code block, a set of test cases with expected pass/fail outcomes, and a diff categorization schema. Without these, the output will be unreliable. In high-risk domains such as security-critical infrastructure, financial systems, or safety-critical software, you must add a human review step after the automated diff analysis. The prompt can flag differences, but it cannot guarantee that a semantic match is behaviorally identical under all conditions.

Before using this prompt, verify that your golden dataset is representative of production inputs and that your test cases cover edge cases, not just happy paths. A common failure mode is treating this prompt as a substitute for integration testing—it compares code structure and test outcomes, but it does not execute the generated code in a production-like environment. Wire this prompt into a CI/CD evaluation harness that runs after code generation and before any deployment gate. If the prompt flags functional differences or test failures, block the release and route the output for human triage. For teams just starting with code generation evaluation, begin with a small golden dataset of 20–50 representative cases and expand as you build confidence in the diff categorization accuracy.

This prompt template is designed for automated code evaluation pipelines where you have a reference implementation and need to assess a generated code sample. It instructs the model to act as a strict diff analyzer, categorizing every difference as cosmetic, semantic, or functional, and integrating test-case pass/fail results. Use this template as the core of your evaluation harness, replacing the square-bracket placeholders with actual values before each run.

codeCopy
You are a strict code diff analyzer. Your task is to compare the [GENERATED_CODE] against the [REFERENCE_CODE] and produce a structured analysis. You must categorize every difference you find.

## Inputs
- **Language:** [PROGRAMMING_LANGUAGE]
- **Generated Code:**
```[PROGRAMMING_LANGUAGE]
[GENERATED_CODE]

• Reference Code:

codeCopy
[REFERENCE_CODE]

• Test Results (if available): [TEST_RESULTS]

Output Schema

Return a single JSON object with the following structure: { "overall_match": boolean, "summary": "string describing the high-level outcome", "differences": [ { "category": "cosmetic" | "semantic" | "functional", "severity": "low" | "medium" | "high", "location": "string describing where the difference occurs (e.g., line numbers, function name)", "generated_snippet": "relevant snippet from generated code", "reference_snippet": "corresponding snippet from reference code", "explanation": "clear explanation of the difference and its impact" } ], "test_integration": { "tests_passed": integer, "tests_failed": integer, "failure_analysis": "string connecting test failures to specific differences, if any" } }

Constraints

• [CONSTRAINTS]
• If no differences exist, return an empty differences array and set overall_match to true.
• Do not hallucinate differences. Only report what is actually present.
• If the generated code is functionally identical but uses different syntax, categorize those differences as cosmetic.
• If test results are provided, you must correlate any test failures with the functional differences you identify.

To adapt this template for your pipeline, start by defining your [CONSTRAINTS]. Common constraints include ignoring whitespace-only changes, treating variable renaming as cosmetic, or flagging any change to a public API signature as high severity. For high-risk domains, add a constraint requiring a human_review_required boolean field in the output when functional differences are detected. Wire this prompt into your evaluation harness by feeding it the output of your code generation step alongside your golden reference implementation. Always validate the JSON output against the expected schema before accepting the result, and log any parse failures for prompt debugging.

The Code Output Diff prompt is designed to sit inside an automated evaluation harness, not as a one-off manual check. The typical integration point is a CI/CD pipeline or an evaluation runner that already has access to the generated code, the reference implementation, and any associated test suites. The harness should call this prompt after code generation completes and before the diff result is accepted as a final evaluation signal. Treat the prompt output as structured evidence, not as a human-readable report—every field in the output schema should be machine-consumable for downstream gating decisions.

Wire the prompt into your application by constructing a request that includes the full generated code, the full reference implementation, and an optional test-results payload. The model should receive these as clearly delimited blocks within the prompt template. On the response side, parse the structured output into a typed object with fields for diff_summary, categorized_differences, semantic_equivalence_judgment, and test_alignment_notes. Validate that every difference entry contains a category field matching your allowed enum (cosmetic, semantic, functional), a severity score, and a location reference. If the output fails schema validation, retry once with a repair prompt that includes the validation error message. If the retry also fails, log the failure and escalate for human review rather than silently accepting a malformed evaluation.

For production reliability, implement a lightweight pre-check before calling the LLM: if the generated code is byte-for-byte identical to the reference, skip the model call entirely and emit a perfect-match result. This saves latency and cost on trivial cases. When test-case results are available, pass them as structured input so the model can cross-reference its diff analysis against actual pass/fail signals. The harness should flag any case where the model claims semantic equivalence but tests fail, or where the model claims functional differences but all tests pass—these mismatches are high-signal events that indicate either test gaps or judge errors. Log every evaluation with the prompt version, model version, input hashes, and output for auditability. For high-risk code generation domains such as security-critical or infrastructure code, always require a human to review diffs categorized as functional before accepting the evaluation as final.