Freeze Gate Behavioral Consistency Check Prompt

This playbook is for QA engineers and AI release managers who need statistical evidence that a system prompt produces stable, repeatable behavior before it is locked for production. The job-to-be-done is not initial prompt design or single-response evaluation; it is the final consistency gate. You use this prompt when you have a candidate system prompt that has passed functional testing and you need to verify that its behavior does not drift across repeated runs, model instances, or minor input variations. The ideal user has a set of representative test inputs, a defined variance threshold, and the authority to block a freeze if the consistency check fails.

Do not use this prompt for evaluating a single model response in isolation, for debugging why a prompt behaves unexpectedly, or for comparing different prompt designs. It is specifically designed to detect flaky behaviors—outputs that change when they shouldn't—by running the same prompt multiple times and measuring output stability. The prompt requires you to define [CONSISTENCY_THRESHOLD] (e.g., 95% pass rate across runs), [VARIANCE_METRIC] (e.g., semantic similarity, exact match, or structural diff), and [INPUT_VARIATIONS] (the set of test cases that will be repeated). Without these, the report cannot produce a reliable pass/fail signal.

Before running this check, ensure you have already completed functional regression testing and edge-case coverage audits. This prompt is the final gate, not a replacement for earlier testing stages. If the consistency check fails, do not proceed with the freeze. Instead, investigate the source of variance—model non-determinism, ambiguous instructions, or temperature settings—and re-run the check after remediation. The output is a structured pass/fail report with statistical evidence; treat a 'fail' as a hard block on the release pipeline.

This template provides the core prompt structure for executing a Freeze Gate Behavioral Consistency Check. It is designed to be dropped into your evaluation harness and configured with your specific system prompt, test inputs, and acceptance thresholds. The prompt instructs the model to act as a QA engineer comparing behavior across repeated runs, model instances, and input variations, producing a structured pass/fail report with statistical evidence. Before using this template, you must have a candidate system prompt that is feature-complete and a representative set of test inputs covering both common and edge-case scenarios.

textCopy
You are a QA engineer performing a final behavioral consistency check before freezing a system prompt. Your task is to evaluate whether the candidate system prompt produces stable, predictable behavior across repeated runs, model instances, and input variations.

## CANDIDATE SYSTEM PROMPT

[SYSTEM_PROMPT]

codeCopy

## TEST CONFIGURATION
- Number of repeated runs per input: [RUN_COUNT]
- Model instances to test: [MODEL_LIST]
- Temperature settings to test: [TEMPERATURE_LIST]
- Acceptable variance threshold (0.0-1.0): [VARIANCE_THRESHOLD]
- Flaky behavior threshold (max inconsistent runs): [FLAKY_THRESHOLD]

## TEST INPUTS
[TEST_INPUTS]

## OUTPUT SCHEMA
Produce a JSON report with the following structure:
{
  "overall_result": "PASS" | "FAIL" | "CONDITIONAL_PASS",
  "summary": {
    "total_inputs_tested": <int>,
    "inputs_passed": <int>,
    "inputs_failed": <int>,
    "flaky_inputs_detected": <int>,
    "average_variance_score": <float>
  },
  "per_input_results": [
    {
      "input_id": "<string>",
      "input_text": "<string>",
      "result": "PASS" | "FAIL" | "FLAKY",
      "variance_score": <float>,
      "inconsistent_runs": <int>,
      "behavior_summary": "<string describing observed behavior range>",
      "failure_mode": "<string or null if passed>"
    }
  ],
  "model_instance_comparison": {
    "cross_model_consistent": <bool>,
    "divergent_models": ["<model names with divergent behavior>"],
    "notes": "<string>"
  },
  "recommendations": [
    "<actionable recommendation strings>"
  ]
}

## CONSTRAINTS
- Compare outputs semantically, not by exact string match. Two responses with the same meaning but different wording should be considered consistent.
- Flag any input where more than [FLAKY_THRESHOLD] runs produce semantically different outputs as FLAKY.
- Flag any input where variance_score exceeds [VARIANCE_THRESHOLD] as FAIL.
- If any model instance produces consistently divergent behavior from others, mark cross_model_consistent as false and list the divergent models.
- Do not hallucinate test results. Only report on inputs you have actually evaluated.
- If the candidate system prompt is ambiguous or underspecified, note this in recommendations rather than failing all tests.

## EVALUATION PROCESS
1. For each test input, simulate [RUN_COUNT] runs across each model instance and temperature setting.
2. Compare outputs within each input group for semantic consistency.
3. Calculate variance scores based on the proportion of divergent outputs.
4. Compare behavior across model instances for the same inputs.
5. Compile the structured report.

To adapt this template, replace each square-bracket placeholder with your specific test configuration. The [SYSTEM_PROMPT] should contain the exact candidate instructions you intend to freeze. [TEST_INPUTS] should be a structured list of representative user inputs, including edge cases, adversarial examples, and common production queries. Set [RUN_COUNT] to at least 5 for statistical significance, and calibrate [VARIANCE_THRESHOLD] based on your application's tolerance for behavioral drift—lower thresholds (0.1-0.2) for high-stakes applications, higher thresholds (0.3-0.5) for creative or open-ended tasks. The [MODEL_LIST] should include all model versions you plan to deploy against, as cross-model consistency is a common failure point. After running the evaluation, treat any CONDITIONAL_PASS result as requiring human review before proceeding with the freeze.

The Freeze Gate Behavioral Consistency Check Prompt is designed to operate as the final automated gate in a CI/CD pipeline for system prompts. It should be invoked after all unit-level regression tests pass but before the release manager signs off on a freeze. The prompt expects a batch of repeated model responses generated from identical inputs across multiple runs, model instances, or temperature settings. The harness must collect these responses, format them into the [INPUT] block with clear run identifiers, and pass any [CONSTRAINTS] such as variance thresholds (e.g., maximum allowed semantic drift score of 0.15) or flaky behavior definitions (e.g., any single-run deviation from the majority output class). The output is a structured pass/fail report with statistical evidence, which the harness must parse and act on.

To implement this reliably, build a Python or TypeScript validation wrapper around the LLM call. The wrapper should: (1) execute the prompt across a pre-configured set of N runs (minimum 5, recommended 10) with controlled temperature and seed settings; (2) collect all responses into a JSON array with run_id, model_instance, and output fields; (3) inject this array into the [INPUT] placeholder along with the [CONSTRAINTS] object specifying variance_threshold and flaky_run_definition; (4) call the model with response_format set to a strict JSON schema matching the expected report structure (fields: overall_result, variance_score, flaky_runs_detected, evidence_summary, recommendation); (5) validate the returned JSON against the schema and retry once on parse failure with a repair prompt; (6) log the full input, output, and validation result to an audit trail. For high-stakes freezes, add a manual review step that surfaces any overall_result: FAIL or variance_score exceeding 80% of the threshold to a QA engineer before blocking the pipeline.

Avoid wiring this prompt directly into an automated freeze without human-readable evidence. The harness should store the raw response batches and the consistency report as artifacts attached to the freeze request. If the model returns a FAIL, the pipeline must halt and notify the release manager with a link to the evidence. Do not proceed with a freeze if the harness encounters more than one retry failure or if the model's output schema does not validate after two attempts—these are signals of an unstable evaluation environment, not a passing prompt. The next step after a passing report is to archive the baseline behavior snapshot and proceed to the stakeholder sign-off prompt.