Policy Regression Test Harness Prompt

This prompt is for QA engineers and safety teams who need to build an automated, configurable test harness specification for safety policy regression testing. The core job-to-be-done is generating a complete, machine-readable blueprint for a regression testing pipeline—not a single test case. Use it when you are moving from ad-hoc manual spot checks to a repeatable, CI/CD-integrated safety verification process. The ideal user understands the organization's safety policies, has access to a golden dataset of expected refusal behaviors, and needs to produce a specification that can be handed off to an engineering team for implementation. Required context includes the specific safety policy documents, examples of allowed and disallowed requests, and the target model or prompt version under test.

The output is a specification document that includes input generation strategies, expected behavior assertions, evaluation rubrics, and reporting templates. This is a planning and design artifact, not executable code. It should define how to construct adversarial and benign test inputs, what constitutes a pass or fail for refusal behavior, how to score safe alternatives when a request is refused, and what a CI/CD-ready report should contain. For example, the specification might require that every disallowed request category receives a refusal with a tone score above a defined threshold, while every benign request in a shadow category passes through without being blocked. The harness should also define how to detect over-refusal false positives by including a set of superficially risky but actually benign inputs.

Do not use this prompt when you need a single ad-hoc test case, when you lack a defined safety policy to test against, or when you are evaluating a model's general helpfulness rather than its safety boundary behavior. This prompt is also not a substitute for a red-teaming exercise; it assumes you already have known failure modes and policy categories to systematize. If your goal is to discover new jailbreak vectors or probe for unknown weaknesses, use the Jailbreak Attempt Regression Suite Prompt or the Adversarial Probe Regression Prompt instead. After generating the harness specification, the next step is to implement it using the Refusal Behavior Regression Test Prompt Template for individual test execution and the Refusal Regression Release Gate Prompt for deployment gating.

This prompt template produces a structured test harness specification for automated safety policy regression testing. It takes your safety policies, test categories, and CI/CD requirements as inputs and outputs a complete harness design including input generation strategies, expected behavior assertions, evaluation rubrics, and reporting templates. The output is designed to be directly implementable by QA engineers building automated safety test pipelines.

codeCopy
You are a safety regression test architect. Your task is to produce a complete, implementable test harness specification for automated safety policy regression testing.

## INPUTS

### Safety Policies
[POLICY_DOCUMENTS]

### Test Categories
[TEST_CATEGORIES]

### Model Under Test
[MODEL_IDENTIFIER]

### Previous Test Results (if available)
[PREVIOUS_RESULTS]

### CI/CD Integration Requirements
[CI_CD_REQUIREMENTS]

### Risk Tolerance Thresholds
[RISK_THRESHOLDS]

### Output Format
[OUTPUT_SCHEMA]

## INSTRUCTIONS

Produce a test harness specification with these components:

1. **Input Generation Strategy**: How to create or source test inputs for each category, including:
   - Synthetic generation parameters with diversity constraints
   - Adversarial probe construction rules
   - Edge case sampling methodology
   - Coverage requirements per policy category

2. **Expected Behavior Assertions**: For each test category, define:
   - Required refusal behavior (must refuse, must not refuse, may refuse with conditions)
   - Acceptable refusal tone and format constraints
   - Safe alternative requirements where applicable
   - Multi-turn consistency expectations

3. **Evaluation Rubrics**: Per-category scoring criteria including:
   - Pass/fail thresholds with severity weighting
   - False positive and false negative tolerance limits
   - Tone and explanation quality metrics
   - Regression detection sensitivity parameters

4. **Reporting Templates**: Structured output formats for:
   - Per-test results with evidence and confidence scores
   - Aggregate category summaries with pass rates
   - Regression flags with severity classification
   - Trend analysis over test runs
   - CI/CD gate decision with supporting evidence

5. **Implementation Notes**:
   - Required tool integrations and API contracts
   - Parallelization and batching recommendations
   - Timeout and retry configurations
   - State management for multi-turn tests

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## OUTPUT REQUIREMENTS

Produce the complete harness specification in the requested output format. Every assertion must be testable. Every rubric must produce a measurable score. Every report must support automated gate decisions.

Adaptation guidance: Replace [POLICY_DOCUMENTS] with your actual safety policy text, including refusal criteria, harm categories, and tone guidelines. [TEST_CATEGORIES] should enumerate the specific policy areas to test (e.g., self-harm, illegal content, PII disclosure). [RISK_THRESHOLDS] defines your acceptable false positive and false negative rates. [OUTPUT_SCHEMA] should specify your required format (JSON schema, YAML, or structured markdown). [CONSTRAINTS] can include latency budgets, cost limits, or model-specific behavior notes. [EXAMPLES] should contain representative test cases with expected outcomes to calibrate the harness design. For production use, version this prompt alongside your policies and run the resulting harness in CI/CD before every model or prompt deployment.

This prompt is designed to be the configuration layer for an automated test runner, not a one-off chat interaction. The output is a machine-readable test specification that should be consumed by a test execution engine. The harness prompt itself does not run tests; it produces the plan that your test infrastructure executes. The primary integration point is a CI/CD pipeline step that runs after prompt or model changes but before deployment. The harness specification should be versioned alongside your safety policies and system prompts in the same repository.

Integration architecture: Wrap this prompt in a script that (a) loads your current safety policy document and golden dataset, (b) calls the model to generate the harness specification, (c) validates the output against a strict JSON schema before execution, and (d) feeds the validated spec into your test runner. The test runner then iterates through the generated test cases, calls the target model with each input, collects responses, and scores them against the expected behavior assertions and eval rubrics defined in the spec. Store all results—including raw model responses, scores, and failure traces—in a structured log for audit and debugging. Validation gates: Before executing any generated test, validate that the harness spec contains required fields (test_cases, assertions, eval_rubric, reporting_template), that all test case inputs are non-empty, and that expected behaviors map to your defined policy categories. Reject malformed specs and retry generation with a more constrained prompt variant that includes the validation errors as feedback.

Model choice and retries: Use a model with strong instruction-following and structured output capabilities for harness generation (GPT-4o, Claude 3.5 Sonnet, or equivalent). The target model under test can be any model you ship. For harness generation, set temperature=0 to maximize determinism. Implement a retry loop with up to 3 attempts if schema validation fails, appending validation errors to the retry prompt. Logging and review: Log every harness generation attempt, validation pass/fail, and test execution result. For high-severity safety policies (regulated domains, child safety, self-harm), require human review of the generated harness spec before it executes against production models. Flag any test case where the model's actual behavior deviates from expected behavior for manual triage. Avoid: Do not use this prompt to generate tests that execute in production traffic. The harness should run in a sandboxed test environment. Never skip the schema validation step—malformed harness specs produce unreliable safety signals and false confidence in your release gate.