Smoke Test Result Verification Prompt

This prompt is designed for release engineers and SRE teams who need a programmatic, auditable gate between a canary deployment and a full production rollout. The core job-to-be-done is converting raw, often verbose, smoke test logs and acceptance criteria into a definitive PASS or FAIL signal that a deployment pipeline can act on. The ideal user is an engineer configuring a post-deployment verification stage in a system like Spinnaker, Argo Rollouts, or a custom GitHub Actions workflow. Required context includes the raw test output, the specific acceptance criteria (e.g., '100% of critical tests must pass, 95% of non-critical'), and deployment metadata such as the target cluster and version tag. Without this structured input, the prompt cannot produce a reliable gating decision.

Do not use this prompt as a replacement for real-time observability or incident response. It is a point-in-time verification step, not a monitoring tool. It is also unsuitable for deployments where the smoke test suite is undefined or where acceptance criteria are purely subjective. The prompt assumes that tests have already executed and produced machine-readable or semi-structured output; it will not execute tests or query live systems. A critical implementation detail is that the prompt includes a coverage gap detection mechanism, flagging when expected test categories (e.g., 'database connectivity', 'cache warm-up') are missing from the results, which prevents a false PASS from an incomplete test run. The output is a structured JSON block containing a top-level verdict, a list of failures, a coverage_gaps array, and a boolean rollback_recommended field, making it directly consumable by downstream pipeline logic.

After integrating this prompt, the next step is to wire its structured output into your deployment orchestrator's conditional logic. If rollback_recommended is true or verdict is FAIL, the pipeline should automatically halt the rollout and trigger a notification to the on-call channel with the failure diagnostics. Avoid the temptation to add complex, multi-step reasoning to this prompt; its value lies in being a fast, deterministic gate. For high-risk deployments, always combine this automated check with a manual approval step in your pipeline, using the prompt's output as the evidence package for the human decision-maker.

This prompt template is designed to be pasted directly into your orchestration layer—whether that's a Python script, a LangChain node, or a custom AI harness. It expects structured smoke test results and a set of acceptance criteria as input. The model's job is to act as a release engineer, comparing the observed behavior against the expected behavior and producing a deterministic, machine-readable verdict. The output includes a clear pass/fail status, a diagnostic breakdown of any failures, and a concrete rollback recommendation. Use this template when you need to close the loop on a deployment verification step before proceeding with a canary promotion or a full rollout.

textCopy
You are a senior release engineer reviewing smoke test results after a deployment.
Your task is to compare the provided smoke test results against the acceptance criteria and produce a structured verification report.

## INPUT
- Smoke Test Results:
[SMOKE_TEST_RESULTS]

- Acceptance Criteria:
[ACCEPTANCE_CRITERIA]

## CONSTRAINTS
- Classify each criterion as PASS, FAIL, or UNTESTED.
- If a criterion is UNTESTED, flag it as a coverage gap and treat it as a FAIL for the overall verdict unless [ALLOW_UNTESTED] is set to true.
- If any criterion FAILS, the overall verdict must be FAIL.
- Do not invent test results. Only use the provided [SMOKE_TEST_RESULTS].
- If the input is malformed or missing, output a FAIL with a diagnostic error.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "overall_verdict": "PASS" | "FAIL",
  "summary": "A one-sentence summary of the outcome.",
  "criteria_results": [
    {
      "criterion": "string",
      "status": "PASS" | "FAIL" | "UNTESTED",
      "evidence": "string explaining the observed vs expected behavior"
    }
  ],
  "coverage_gaps": ["list of untested criteria"],
  "failure_diagnostics": "A detailed explanation of the root cause of the first failure, or null if passed.",
  "rollback_recommendation": "ROLLBACK" | "NO_ROLLBACK" | "INVESTIGATE",
  "rollback_reasoning": "A concise justification for the rollback recommendation."
}

## RISK_LEVEL
[HIGH_RISK_ACTION]

## EXAMPLES
[FEW_SHOT_EXAMPLES]

To adapt this template for your environment, replace the square-bracket placeholders with data from your test runner and deployment system. The [SMOKE_TEST_RESULTS] should be a structured blob, ideally JSON, containing the output of your automated smoke tests. The [ACCEPTANCE_CRITERIA] should be a list of explicit, verifiable conditions. The [ALLOW_UNTESTED] flag lets you control whether incomplete test coverage is a hard failure. The [HIGH_RISK_ACTION] placeholder should be set to true if the deployment is in a production environment, which will instruct the model to apply stricter scrutiny. The [FEW_SHOT_EXAMPLES] placeholder is critical for calibrating the model's judgment on edge cases; provide at least one example of a clean pass and one example of a failure with a clear rollback trigger. After pasting this template, always run it against a known set of golden test cases to validate the output schema and the correctness of the verdict logic before trusting it in a live pipeline.

The Smoke Test Result Verification Prompt is designed to sit at the end of a release pipeline, consuming raw test output and acceptance criteria to produce a structured pass/fail verdict. The implementation harness must treat the model's output as a decision-support artifact, not an autonomous gate. The primary integration point is a post-smoke-test job in your CI/CD system (e.g., a GitHub Actions workflow, a Jenkins stage, or a GitLab CI job) that collects test results, assembles the prompt, calls the model, and then validates the response before acting on it. The harness should never automatically roll back or promote a release based solely on the raw model response; a validation layer and, for high-risk environments, a human-approval step must sit between the model output and the deployment decision.

To wire this in, start by defining a strict output schema that the prompt must follow—typically a JSON object with verdict (enum: PASS, FAIL, INCONCLUSIVE), failure_diagnostics (array of objects with test_name, expected, actual, severity), and rollback_recommendation (boolean). After receiving the model response, run a structural validator that checks for schema compliance, enum membership, and required fields. If validation fails, implement a retry loop (maximum 2 retries) that feeds the validation error back into the prompt as additional context. If the verdict is INCONCLUSIVE or the retry budget is exhausted, escalate to a human review queue with the raw test output, the partial model analysis, and a structured handoff note. For model choice, prefer a model with strong JSON-mode support and a low rate of schema drift; temperature should be set to 0 or near-zero to maximize deterministic behavior on pass/fail classification. Log every prompt version, raw response, validation result, and final decision to an audit trail for post-release governance.

A critical failure mode in this harness is test coverage gap detection. The prompt includes instructions to flag missing test categories, but the harness should independently cross-reference the executed test list against a known coverage baseline (e.g., a checklist of required smoke test categories for the service). If the model flags a gap, or if the harness detects a gap the model missed, the verdict should be forced to FAIL or INCONCLUSIVE regardless of individual test results. Another failure mode is the model hallucinating test results that were not in the input. Mitigate this by implementing a grounding check: extract all test names from the model's failure_diagnostics and confirm they exist in the input test results. Any hallucinated test name should trigger a validation failure and retry. For high-risk releases, require a human to explicitly approve the final verdict before the pipeline proceeds to the next stage. The human-approval interface should display the model's structured output, the raw test results, and any validation warnings in a single review pane.

Next, integrate this harness into your pipeline's promotion logic. A PASS verdict with successful validation can automatically unblock the next deployment stage. A FAIL verdict should block promotion and, if rollback_recommendation is true, trigger a separate rollback workflow that also requires human confirmation. An INCONCLUSIVE verdict should pause the pipeline and notify the on-call release engineer. Avoid the temptation to skip the validation layer or the human gate for low-risk environments; even non-critical services benefit from an audit trail that captures why a release was promoted or blocked. The harness code itself should be versioned alongside the prompt template, and any change to the prompt, schema, or validation rules should trigger a regression run against a golden dataset of known smoke test scenarios before the updated harness reaches production.