Output Contract Assertion Prompt for CI/CD Gates

This prompt is designed for test architects and platform engineers who need to programmatically gate an LLM's output before it reaches downstream systems. It functions as a programmable contract checker, accepting raw LLM output and a natural-language contract that describes required fields, types, value ranges, and constraints. The primary job-to-be-done is to produce a machine-readable pass/fail verdict with a detailed violation list, enabling automated promotion or rollback decisions in a CI/CD pipeline. The ideal user is someone who already has a deterministic JSON Schema validator in place but has found it insufficient for rules requiring reasoning, such as conditional logic, semantic constraints, or complex business rules that cannot be expressed in a static schema.

Use this prompt when your validation logic requires an LLM's reasoning capabilities. For example, a contract might state 'the discount_percentage must be 0 if the customer_tier is not premium,' or 'the summary field must not contradict the detailed_findings array.' These cross-field, semantic, and conditional checks are brittle to encode in traditional code but are natural to express in a prompt. The prompt is best deployed as a synchronous gate in a non-real-time batch processing or pre-commit CI step, where a few seconds of latency is acceptable. You should provide the raw LLM output, the natural-language contract, and an expected output schema for the verdict itself to ensure the assertion is machine-parseable.

Do not use this prompt for simple type checking where a deterministic validator is sufficient and more cost-effective. If your contract is fully expressible as a JSON Schema, a standard library will be faster, cheaper, and immune to LLM variability. Avoid this prompt in real-time streaming contexts where the added latency of an extra LLM call would degrade the user experience. Finally, do not use this as a primary safety guard for high-risk content; it should be one layer in a defense-in-depth strategy that includes human review for critical actions. The next step is to define your contract in precise, unambiguous natural language and prepare a set of passing and failing test cases to calibrate the prompt's strictness before wiring it into your pipeline.

This prompt template is the executable core of your output contract assertion gate. It is designed to be dropped directly into a test harness, an evaluation script, or a CI/CD step. The prompt instructs the model to act as a strict validator, comparing an LLM-generated output against a custom contract you define. It produces a machine-readable JSON verdict that your pipeline can parse to decide whether to promote or block a change. The template uses square-bracket placeholders for all variable parts—your specific contract, the output under test, and any optional context—so you can adapt it without rewriting the instruction logic.

textCopy
You are an output contract validator for a CI/CD pipeline. Your job is to check whether the provided LLM output satisfies a custom contract. You must be strict, deterministic, and produce only the specified JSON response.

# CONTRACT
[CONTRACT]

# OUTPUT UNDER TEST
[OUTPUT]

# OPTIONAL CONTEXT
[CONTEXT]

# INSTRUCTIONS
1. Parse the CONTRACT. It defines required fields, their expected types, allowed value ranges, and any cross-field constraints.
2. Examine the OUTPUT UNDER TEST.
3. For each requirement in the CONTRACT, determine if the OUTPUT satisfies it.
4. If the CONTRACT references OPTIONAL CONTEXT, use that context to resolve any ambiguity, but do not relax the contract requirements.
5. Produce a single JSON object with the following schema:

{
  "verdict": "PASS" | "FAIL",
  "violations": [
    {
      "field": "string (path to the violating field, e.g., 'user.address.zip')",
      "rule": "string (the specific contract rule that was violated)",
      "expected": "string (what the contract required)",
      "actual": "string (what the output provided)",
      "severity": "BLOCKING" | "WARNING"
    }
  ],
  "summary": "string (one-sentence summary of the validation result)"
}

# RULES
- If the OUTPUT satisfies all contract requirements, return verdict PASS with an empty violations array.
- If any BLOCKING violation exists, verdict must be FAIL.
- If only WARNING violations exist, verdict may still be PASS, but you must list all warnings.
- Do not hallucinate violations. Only report what is actually wrong.
- If the OUTPUT is malformed or unparseable against the contract, return a single BLOCKING violation with field "output", rule "parse_error", and the actual value as the raw output string.
- Do not include any text outside the JSON object.

To adapt this template, replace [CONTRACT] with a structured description of your requirements. This can be a JSON Schema, a list of field-level rules, or a natural language specification of required fields, types, and constraints. Replace [OUTPUT] with the actual LLM response you are validating. The [CONTEXT] placeholder is optional—use it to provide source documents, user input, or other evidence the validator should reference when checking grounding-dependent rules. After pasting, wire the prompt into your assertion harness so that the returned JSON verdict field controls a pass/fail gate. For high-risk domains, always include a human review step before blocking a release based solely on this prompt's output.

This prompt is designed to be the final gate in a CI/CD pipeline before a prompt change is promoted. It should be called after the prompt under test generates its output, receiving both the output and the contract definition as input. The assertion prompt itself must be treated as a deterministic test oracle: it should run with a low temperature (0.0–0.1) and a constrained output format to minimize variance in pass/fail decisions. In a typical pipeline, a test runner (such as pytest or a custom GitHub Actions workflow) executes a set of test cases, collects the LLM outputs, and then calls this assertion prompt for each output-contract pair. The assertion result is parsed and converted into a binary gate: any FAIL verdict blocks the release.

To integrate this into a CI/CD system, wrap the prompt in a validation harness that enforces a strict contract on the assertion output itself. The harness should parse the JSON response and verify that it contains the required fields (verdict, violations, checked_fields). If the assertion prompt returns malformed JSON or an unexpected structure, the harness must treat this as a test infrastructure failure, not a pass. Implement a retry loop with a maximum of 2 attempts for malformed responses, using a backoff of 1 second. Log every assertion result—including the input contract, the output under test, the raw assertion response, and the parsed verdict—as a structured artifact attached to the CI run. For high-risk domains such as healthcare or finance, add a manual approval step that requires a human to review any FAIL verdict before the pipeline can be unblocked, and archive all assertion artifacts for audit purposes.

When choosing a model for the assertion prompt, prefer a model that is strong at instruction-following and structured output generation, such as gpt-4o or claude-3.5-sonnet. Avoid using the same model that generated the output under test, as this can mask systematic errors. The assertion prompt should be versioned alongside the prompt it tests, and any change to the assertion prompt itself must pass its own regression suite before being used in a gate. Start by running this assertion against a golden dataset of known passing and failing outputs to calibrate your thresholds and confirm that the assertion prompt correctly identifies contract violations without excessive false positives.