CI/CD Prompt Gate Integration Prompt

This prompt is for DevOps and MLOps engineers who need to embed automated prompt evaluation directly into a CI/CD release pipeline. It produces a machine-readable gate configuration that defines when prompt tests should run, what constitutes a pass or fail, how artifacts should be versioned, and who gets notified. Use this when you are moving from ad-hoc prompt testing to a repeatable, automated gate that blocks or promotes prompt changes based on evidence.

This prompt assumes you already have a test suite, golden dataset, and evaluation harness defined. It does not design those tests; it generates the pipeline integration rules that make them a formal release gate. The output is a configuration artifact—typically YAML or JSON—that can be consumed by your CI/CD system (GitHub Actions, GitLab CI, Jenkins) to trigger evaluation jobs, collect results, compare against thresholds, and execute promotion or rollback actions. It also generates notification templates for Slack, email, or incident management tools so the right people know when a prompt change passes or fails the gate.

Do not use this prompt if you are still building your first test suite or golden dataset. Start with the Golden Dataset Construction Prompt and Regression Test Harness Prompt in this pillar. Also avoid this prompt if your release process requires human review of every prompt change without automation—this prompt is designed for teams ready to formalize automated gates. For high-risk domains (healthcare, finance, legal), the generated configuration should always include a manual approval step before production promotion, even if the automated tests pass.

This prompt template is designed for DevOps engineers and AI platform teams who need to embed automated prompt quality checks directly into their CI/CD pipelines. It produces a complete gate configuration—including test triggers, pass/fail actions, artifact versioning, and notification rules—that can be adapted to your specific orchestration tool. The output is a structured specification, not a runnable script, giving you the blueprint to implement the gate in your environment of choice.

textCopy
You are a CI/CD pipeline architect specializing in LLM release engineering. Your task is to generate a gate configuration for integrating a prompt evaluation suite into a release pipeline.

## INPUTS
- Pipeline Tool: [PIPELINE_TOOL, e.g., GitHub Actions, GitLab CI, Jenkins]
- Prompt Artifact Path: [PROMPT_ARTIFACT_PATH]
- Eval Harness Command: [EVAL_HARNESS_COMMAND]
- Golden Dataset Path: [GOLDEN_DATASET_PATH]
- Pass Thresholds: [PASS_THRESHOLDS, e.g., "exact_match >= 0.95, semantic_similarity >= 0.90, schema_validity = 1.0"]
- Notification Channels: [NOTIFICATION_CHANNELS, e.g., Slack #prompt-releases, email alerts]
- Environment: [ENVIRONMENT, e.g., staging, production]
- Rollback Trigger Rules: [ROLLBACK_TRIGGERS, e.g., "error_rate > 0.05 OR critical_failure_count > 0"]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "gate_name": "string",
  "triggers": {
    "on_push": { "branches": ["string"] },
    "on_pull_request": { "branches": ["string"] },
    "manual_approval_required": boolean
  },
  "stages": [
    {
      "name": "string",
      "description": "string",
      "command": "string",
      "timeout_seconds": number,
      "retry_count": number,
      "continue_on_failure": boolean
    }
  ],
  "pass_criteria": {
    "metrics": [
      {
        "name": "string",
        "threshold": "string",
        "operator": "gte | lte | eq",
        "weight": number
      }
    ],
    "aggregation_rule": "all | any | weighted_majority"
  },
  "actions": {
    "on_pass": {
      "artifact_versioning": "string",
      "promotion_action": "string",
      "notifications": ["string"]
    },
    "on_fail": {
      "rollback_action": "string",
      "incident_creation": "string",
      "notifications": ["string"],
      "manual_override_instructions": "string"
    }
  },
  "smoke_test": {
    "enabled": boolean,
    "test_count": number,
    "quick_fail_threshold": "string"
  },
  "canary_deployment": {
    "enabled": boolean,
    "traffic_percentage": number,
    "observation_period_minutes": number,
    "auto_promotion_delay_minutes": number
  }
}

## CONSTRAINTS
- The gate must fail closed: any unrecoverable error in the eval harness should trigger a failure.
- Include a smoke test stage that runs a small subset of critical examples before the full suite.
- Canary deployment hooks must include an observation period with automatic rollback if error rates spike.
- All notifications must include a direct link to the failing test run and a summary of which thresholds were breached.
- The rollback action must preserve the previous prompt artifact and revert the active version.
- Do not invent specific tool syntax; use generic command placeholders that can be adapted.

After copying the template, replace each square-bracket placeholder with your actual pipeline details. The EVAL_HARNESS_COMMAND should point to your existing test runner—this prompt assumes you already have a regression test suite and golden dataset in place. If you are using a pipeline tool with native YAML or JSON configuration, map the output stages and actions to that tool's syntax. For high-risk production deployments, ensure the manual_approval_required flag is set to true and that the rollback trigger rules are conservative enough to catch regressions before they affect a large user base. Test the gate in a staging environment first with a known-bad prompt to verify that the failure path works as expected.

This prompt is designed to be called programmatically from within a CI/CD pipeline (GitHub Actions, GitLab CI, Jenkins, etc.) after a prompt version change is detected and regression tests have completed. The prompt expects structured inputs—test results, deployment context, and risk parameters—and returns a gate decision that downstream jobs can act on. Do not call this prompt in isolation without first running your regression test suite; the gate configuration it produces is only as reliable as the evaluation data fed into it.

Wire the prompt into your pipeline as a dedicated gate stage. Pass the [TEST_RESULTS] as a JSON blob containing per-example pass/fail counts, semantic drift scores, and schema validation results from your test harness. Include [DEPLOYMENT_CONTEXT] with the target environment (staging, canary, production), the prompt version identifier, and the previous stable version for comparison. The [RISK_PROFILE] should specify the blast radius (e.g., 'customer-facing-chat', 'internal-tool') and any regulatory constraints. On the output side, parse the gate decision (proceed, hold, rollback) and use it to control pipeline flow. If the decision is hold, route to a manual approval step with the generated justification and recommended actions. If rollback, trigger your automated revert workflow and post the incident documentation template to your alerting channel.

Build validation around the prompt's output before acting on it. Validate that the returned JSON matches the expected schema: a top-level decision field with one of three allowed enum values, a confidence score between 0 and 1, and a non-empty justification array. If the output fails schema validation, retry once with the validation error appended to the [CONSTRAINTS] field. If the retry also fails, default to hold and escalate to the on-call engineer. Log every gate evaluation—including the prompt version, test run ID, decision, and raw output—to your observability platform for audit trails and future prompt debugging. For high-risk deployments (customer-facing, regulated domains), require human approval even when the gate returns proceed, and attach the full evaluation evidence to the approval request.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models that may hallucinate gate decisions or drop required fields. Set temperature=0 for deterministic outputs. If your pipeline runs frequently, consider caching the prompt prefix (the system instructions and output schema) to reduce latency and cost. The gate decision should take under 5 seconds in the common case; if latency exceeds 10 seconds, implement a timeout that defaults to hold and logs a warning. Do not use this prompt as a substitute for actual test execution—it is a decision layer on top of test results, not a test runner itself.