Golden Dataset Pass Rate Gate Prompt

This prompt is designed for ML engineers and platform teams who need a deterministic, evidence-based gate before promoting a prompt from staging to production. The core job-to-be-done is comparing a new prompt version's outputs against a curated golden dataset of known inputs and expected outputs. It produces a structured pass-rate report that includes per-category breakdowns, regression detection against a prior baseline version, and failure clustering. Use this when your release process requires a quantitative minimum pass-rate threshold to be met automatically, reducing the risk of subjective or premature promotions.

To use this prompt effectively, you must have a stable, versioned golden dataset where each record contains an input, a ground-truth expected output, and optional metadata like category tags. The prompt is wired to compare the [NEW_PROMPT_OUTPUTS] against [GOLDEN_EXPECTED_OUTPUTS] and, if provided, [BASELINE_PROMPT_OUTPUTS] to detect regressions. It enforces a [MINIMUM_PASS_RATE] and can cluster failures by [CATEGORY_FIELD]. A typical implementation involves running both the new and baseline prompts across the entire golden dataset offline, then feeding the collected outputs into this evaluation prompt. The output is a machine-readable report that can be parsed by a CI/CD pipeline to automatically block or allow progression.

Do not use this prompt when you lack a stable golden dataset, as the gate is only as reliable as the ground truth it tests against. It is also unsuitable for evaluating non-deterministic creative tasks where a single correct answer does not exist, or for scenarios requiring real-time human judgment calls on nuance, safety, or tone. If your evaluation requires an LLM judge to assess semantic similarity rather than exact matching, you should pair this gate with a separate semantic evaluation step before relying on its pass-rate signal. Finally, avoid using this prompt as the sole gate for high-risk domains like healthcare or finance; always supplement it with human review and evidence-grounding checks.

This is the core prompt template for the Golden Dataset Pass Rate Gate. It is designed to be wired into an evaluation harness that supplies a versioned golden dataset, a candidate prompt, and a set of pass/fail criteria. The template instructs the model to act as an automated evaluator, comparing the candidate's outputs against expected references, detecting regressions from a prior baseline version, and clustering failures for root-cause analysis. Use this template as the starting point for your CI/CD gate; copy it directly and replace the square-bracket placeholders with your specific inputs.

textCopy
You are an automated prompt evaluation system. Your task is to assess a candidate prompt against a golden dataset and produce a structured pass-rate report.

## INPUTS
- Golden Dataset: [GOLDEN_DATASET]
- Candidate Prompt: [CANDIDATE_PROMPT]
- Baseline Prompt Version (for regression detection): [BASELINE_PROMPT]
- Evaluation Criteria: [EVALUATION_CRITERIA]
- Minimum Pass Rate Threshold: [MINIMUM_PASS_RATE]
- Output Schema: [OUTPUT_SCHEMA]

## INSTRUCTIONS
1. For each example in the Golden Dataset, execute the Candidate Prompt with the provided input and generate an output.
2. Compare the candidate output against the expected reference output using the Evaluation Criteria.
3. For each example, assign a PASS or FAIL status with a brief justification.
4. If a Baseline Prompt Version is provided, execute it on the same inputs and compare its outputs to the candidate's. Flag any example where the candidate fails but the baseline passed as a REGRESSION.
5. Cluster all failures into categories based on common failure modes (e.g., format error, factual error, refusal, missing citation).
6. Calculate the overall pass rate as (PASS count / Total examples).
7. Compare the overall pass rate to the Minimum Pass Rate Threshold and assign a GATE_STATUS of PASS or FAIL.

## CONSTRAINTS
- Do not modify the Candidate Prompt or Golden Dataset.
- If an expected output is missing or ambiguous, flag the example as UNEVALUABLE and exclude it from the pass rate calculation.
- Report exact counts; do not round percentages in intermediate steps.
- If the Baseline Prompt Version is not provided, skip regression detection and note its absence.

## OUTPUT
Return a single JSON object conforming to the Output Schema. Do not include any text outside the JSON object.

To adapt this template for your pipeline, replace [GOLDEN_DATASET] with a serialized JSON array of input-output pairs, each with a unique example_id. [CANDIDATE_PROMPT] should be the full system and user prompt template you are evaluating. [EVALUATION_CRITERIA] can be a list of semantic or structural rules, such as "output must be valid JSON with all required fields" or "the answer must cite a source." The [OUTPUT_SCHEMA] placeholder should contain your expected JSON Schema for the report, including fields for pass_rate, gate_status, regression_count, and failure_clusters. If you are not running a regression comparison, set [BASELINE_PROMPT] to an empty string and the model will skip that step. Always set [MINIMUM_PASS_RATE] to a numeric value between 0.0 and 1.0 to enable the automated gate decision.

The Golden Dataset Pass Rate Gate Prompt is designed to be the final quantitative checkpoint in a prompt release pipeline. It consumes a curated dataset of inputs paired with expected outputs, runs the candidate prompt against every example, and produces a structured pass-rate report. The harness must enforce a minimum overall pass rate and per-category thresholds before allowing promotion. This prompt is not a one-off manual check; it is a programmatic gate that should be called by an evaluation runner, a CI/CD job, or a model release orchestrator. The harness is responsible for dataset versioning, metric collection, and the binary promote/hold decision.

To implement this gate, you need five components: a versioned golden dataset, an evaluation runner, the prompt template, a comparison function, and a threshold configuration. The golden dataset should be stored in a repository with explicit version tags so every gate run is reproducible. Each record must include an input field, an expected_output field, and a category label for per-category breakdowns. The evaluation runner iterates over the dataset, sends each input to the model with the candidate prompt, and collects the raw outputs. The comparison function then judges each output against the expected output using the criteria defined in the prompt (exact match, semantic equivalence, or structured field comparison). The threshold configuration defines the minimum overall pass rate (e.g., 0.95) and per-category minimums (e.g., 0.90 for edge_cases). If any threshold is violated, the gate fails and the release is blocked.

For high-risk domains, add a human review checkpoint when the pass rate falls within a warning band (e.g., 0.90–0.95). The harness should log every failure with the input, expected output, actual output, and category for post-run analysis. Use the failure clustering output from the prompt to detect systemic regressions—if a single failure mode accounts for more than 20% of failures, flag it for root-cause investigation before retrying. Implement retry logic with exponential backoff only for transient model API errors, not for evaluation failures. Store all run artifacts (dataset version, prompt version, model identifier, pass-rate report, and failure logs) in an immutable artifact store for auditability. Finally, wire the harness exit code to your CI/CD pipeline: exit 0 for pass, exit 1 for failure, and exit 2 for infrastructure errors that require operator attention.