Behavioral Policy A/B Test Prompt

This prompt is for product and AI engineering teams who need to compare two versions of a system-level behavioral policy in a live application. Use it when you have a specific hypothesis about how a policy change will affect user-facing metrics, safety outcomes, or model performance. The prompt produces a complete experiment design document, including a hypothesis, primary and guardrail metrics, traffic allocation, duration, and statistical significance thresholds. It also defines automatic shutdown criteria if safety metrics degrade.

Do not use this prompt for comparing model versions, testing retrieval pipelines, or evaluating non-behavioral prompt changes. It assumes you already have two policy variants drafted and a production system capable of splitting traffic and logging metrics. The prompt is designed for behavioral policies—rules that govern how an assistant responds, refuses, discloses uncertainty, or escalates—not for changes to factual knowledge, tool schemas, or persona definitions. If you are testing a new system prompt that changes both behavior and capabilities, isolate the behavioral component first or use a broader regression test prompt instead.

Before running this prompt, confirm that your metrics pipeline can reliably capture the outcomes you intend to measure. The prompt will ask you to specify primary metrics (e.g., task completion rate, user satisfaction score) and guardrail metrics (e.g., policy violation rate, refusal rate on safe requests). If your logging infrastructure cannot separate these signals by experiment group, the resulting experiment design will be unusable. Also ensure that your traffic-splitting mechanism supports consistent user assignment—users who bounce between policy variants will produce noisy data and invalidate statistical conclusions.

This prompt template is designed to be pasted directly into your LLM interface. It instructs the model to function as a rigorous experiment design engine, taking your specific policy variants, success metrics, and guardrail constraints as input. The model's sole job is to produce a complete, structured A/B test plan in JSON format. Before using this prompt, you must have clearly defined the two behavioral policies you intend to compare, the primary metric you aim to improve, and the safety metrics you cannot degrade. This prompt is not a substitute for a data scientist's review; it is a drafting tool that accelerates the creation of a test plan that must be validated before any traffic is allocated.

textCopy
You are an experiment design engine for AI behavioral policies. Your output must be a single, valid JSON object conforming to the schema below. Do not include any text outside the JSON object.

## Inputs
- **Policy A (Control):** [POLICY_A_DESCRIPTION]
- **Policy B (Treatment):** [POLICY_B_DESCRIPTION]
- **Primary Success Metric:** [PRIMARY_METRIC]
- **Guardrail Metrics:** [GUARDRAIL_METRICS]
- **Traffic Allocation:** [TRAFFIC_ALLOCATION]
- **Context:** [EXPERIMENT_CONTEXT]

## Output Schema
{
  "experiment_name": "string",
  "hypothesis": {
    "null_hypothesis": "string",
    "alternative_hypothesis": "string",
    "rationale": "string"
  },
  "design": {
    "control_variant": "Policy A",
    "treatment_variant": "Policy B",
    "traffic_allocation": {
      "control_percent": number,
      "treatment_percent": number
    },
    "randomization_unit": "string",
    "target_duration_days": number
  },
  "metrics": {
    "primary": {
      "name": "string",
      "definition": "string",
      "direction": "increase | decrease"
    },
    "guardrails": [
      {
        "name": "string",
        "definition": "string",
        "acceptable_boundary": "string"
      }
    ]
  },
  "analysis_plan": {
    "statistical_test": "string",
    "significance_level": number,
    "minimum_detectable_effect": "string",
    "interim_analysis_frequency": "string"
  },
  "safety_rules": {
    "automatic_shutdown_conditions": ["string"],
    "degradation_threshold": "string"
  },
  "limitations_and_risks": ["string"]
}

## Constraints
- [CONSTRAINT_1]
- [CONSTRAINT_2]
- [CONSTRAINT_3]

To adapt this template, replace the square-bracket placeholders with your specific details. For [POLICY_A_DESCRIPTION] and [POLICY_B_DESCRIPTION], provide the exact system instructions or behavioral rules for each variant. For [PRIMARY_METRIC], specify a measurable outcome like 'user task completion rate' or 'support escalation deflection'. [GUARDRAIL_METRICS] are the safety and quality metrics that must not degrade, such as 'refusal rate on safe prompts' or 'hallucination frequency'. Use the [CONSTRAINTS] section to enforce business rules, like 'experiment duration cannot exceed 14 days' or 'traffic allocation must be a 50/50 split'. The model will use these constraints to shape the design and analysis_plan fields. After generating the JSON, you should validate it against the schema programmatically before using it to configure your feature flag or experimentation platform.

This prompt is most effective with models that have strong instruction-following and structured output capabilities. For high-stakes policy changes that affect user safety or regulated workflows, the generated experiment design must be reviewed by a qualified data scientist or AI safety engineer. The safety_rules.automatic_shutdown_conditions field is critical; ensure the conditions are concrete and can be monitored in real time. A vague condition like 'if things look bad' will not protect users. Instead, the prompt should be adapted to produce a condition like 'if the 7-day moving average of the safety refusal rate for Policy B exceeds the control rate by more than 5 percentage points, with p < 0.05'. The value of this prompt is in producing a consistent, auditable experiment document that forces explicit decisions about risk tolerance before any code is deployed.

This prompt is not a one-off query; it is a design document generator meant to be called by an experiment management system or a CI/CD pipeline. The output is a structured experiment plan that downstream systems consume to configure traffic splitting, metric collection, and automatic shutdown logic. Treat the prompt as the first stage in a multi-step A/B testing workflow where the model proposes the experiment design and a human or automated gate approves it before any traffic is allocated.

Wire the prompt into an application by first assembling the required inputs: [POLICY_VARIANT_A], [POLICY_VARIANT_B], [PRIMARY_METRICS], [SAFETY_METRICS], [TRAFFIC_ALLOCATION], [MINIMUM_DETECTABLE_EFFECT], and [EXPECTED_TRAFFIC_VOLUME]. These should be pulled from a feature flag system, a policy registry, or a configuration store—never hardcoded. After the model returns the experiment design, validate the output against a strict schema that checks for required fields: hypothesis, metrics (with primary and safety arrays), traffic_allocation (with control and treatment percentages), duration_days, statistical_significance_threshold, and auto_shutdown_conditions. Reject any output missing these fields and retry with a repair prompt that includes the validation errors. Log every generated plan, validation result, and human approval decision to an audit table with the prompt version, model version, and timestamp.

The most critical integration point is the auto-shutdown guardrail. The prompt produces auto_shutdown_conditions as structured rules, such as 'if safety_metric_X degrades by more than Y% with p < 0.05, halt the experiment.' Your application must parse these conditions and wire them into your metrics pipeline as automated checks that run on every data refresh. Do not rely on a human noticing a safety degradation in a dashboard. If your metrics pipeline cannot evaluate the shutdown conditions programmatically, the experiment must not start. After the experiment concludes, feed the results back into a follow-up analysis prompt—not this one—to produce a recommendation to ship, iterate, or rollback. Avoid using this prompt for real-time decisioning; it is a planning tool, not an operational controller.