Confidence Threshold A/B Test Configuration Prompt

This prompt is for safety engineers and ML platform operators who need to run controlled experiments on refusal thresholds in a production AI system. The job-to-be-done is generating a deterministic, auditable configuration that splits traffic between two or more risk threshold arms—such as a conservative baseline and an experimental setting—while preserving consistent logging for downstream comparison. The ideal user already has a safety classifier producing calibrated risk scores and needs to measure the causal impact of changing the refusal threshold on metrics like refusal rate, user satisfaction, and safety incident counts.

Use this prompt when you have a defined safety policy, a risk scoring pipeline, and a routing layer that can consume a threshold configuration. It is appropriate for gradual rollouts, dark launches, and pre-release safety tuning. Do not use this prompt when you lack calibrated risk scores, when the safety classifier itself is still under active development, or when the product surface does not support dynamic threshold routing. This prompt is not a replacement for a safety classifier evaluation; it assumes the classifier's confidence scores are meaningful enough to act on. If you are still building the classifier, start with the Safety Classification Confidence Scoring Prompt Template instead.

The output is a machine-readable configuration—typically JSON—that maps traffic percentages to threshold values, defines logging schemas for each arm, and specifies the experiment duration and success criteria. Before deploying the generated configuration, validate that the traffic splits sum to 100%, that each threshold value falls within the classifier's supported range, and that the logging schema captures the fields required for your statistical analysis. For high-risk domains such as regulated content or child safety, require human approval on the experiment plan before any traffic is routed to a less conservative arm. After deployment, monitor for threshold boundary crowding and distribution shifts using the companion Safety Score Distribution Monitoring Prompt.

This prompt template produces a structured configuration object that defines two experimental arms—typically a control arm using the current refusal threshold and a treatment arm using a candidate threshold. It specifies traffic allocation percentages, logging requirements, metrics to capture, and the statistical comparison method. The output is designed to be consumed directly by a feature flag system or an experiment platform's configuration API. Use this when you need to move from ad-hoc threshold tuning to a controlled, measurable experiment that lets you compare refusal rates, user satisfaction signals, and safety incident counts between threshold settings before committing to a change.

codeCopy
You are a safety experiment configuration generator for production AI systems. Your task is to produce a complete A/B test configuration for comparing two refusal risk thresholds. Output only valid JSON matching the [OUTPUT_SCHEMA]. Do not include commentary outside the JSON object.

## INPUTS
- Current production threshold: [CURRENT_THRESHOLD]
- Candidate threshold to evaluate: [CANDIDATE_THRESHOLD]
- Risk score range: [RISK_SCORE_RANGE]
- Harm categories affected: [HARM_CATEGORIES]
- Traffic allocation for treatment arm: [TREATMENT_ALLOCATION_PERCENT]
- Minimum sample size per arm: [MIN_SAMPLE_SIZE]
- Experiment duration in days: [EXPERIMENT_DURATION_DAYS]
- Statistical significance level: [SIGNIFICANCE_LEVEL]
- Logging requirements: [LOGGING_REQUIREMENTS]
- Escalation rules for high-risk cases: [ESCALATION_RULES]

## CONSTRAINTS
- The control arm must use [CURRENT_THRESHOLD] exactly.
- The treatment arm must use [CANDIDATE_THRESHOLD] exactly.
- Traffic allocation must sum to 100% across all arms.
- Requests with risk scores above [MAX_OVERRIDE_THRESHOLD] must always be refused regardless of arm assignment.
- Requests with risk scores below [MIN_OVERRIDE_THRESHOLD] must always be allowed regardless of arm assignment.
- Logging must capture all fields specified in [LOGGING_REQUIREMENTS] for every request in both arms.
- The configuration must include a kill switch condition that halts the experiment if the treatment arm's safety incident rate exceeds [SAFETY_INCIDENT_LIMIT].

## OUTPUT_SCHEMA
{
  "experiment_id": "string, unique identifier for this experiment",
  "description": "string, human-readable summary of what is being tested",
  "arms": [
    {
      "arm_name": "string, 'control' or descriptive label",
      "threshold": number,
      "traffic_percent": number,
      "is_control": boolean
    }
  ],
  "override_rules": {
    "always_refuse_above": number,
    "always_allow_below": number
  },
  "metrics": [
    {
      "metric_name": "string",
      "description": "string",
      "aggregation": "string, e.g., 'rate', 'count', 'mean', 'p95'",
      "primary": boolean
    }
  ],
  "statistical_test": {
    "method": "string, e.g., 'two-proportion z-test', 't-test', 'chi-squared'",
    "significance_level": number,
    "correction": "string, e.g., 'bonferroni', 'none'"
  },
  "logging_schema": {
    "required_fields": ["string"],
    "optional_fields": ["string"]
  },
  "kill_switch": {
    "condition": "string, e.g., 'treatment_safety_incident_rate > 0.02'",
    "action": "string, e.g., 'halt_experiment_and_route_all_to_control'"
  },
  "duration_days": number,
  "minimum_sample_per_arm": number
}

## EXAMPLES
Example input:
- Current threshold: 0.65
- Candidate threshold: 0.72
- Risk score range: 0.0 to 1.0
- Harm categories: ["violence", "hate_speech", "self_harm"]
- Treatment allocation: 10%
- Minimum sample size: 5000
- Duration: 14 days
- Significance level: 0.05
- Logging: ["request_id", "risk_score", "arm", "refusal_decision", "harm_category", "user_appeal"]
- Escalation rules: "scores above 0.95 always escalate to human review"

Generate the configuration now for the provided inputs.

After copying this template, replace every square-bracket placeholder with concrete values from your safety system's current configuration and the experiment you intend to run. The override_rules section is critical for safety: it ensures that extremely high-risk requests are always refused and extremely low-risk requests are always allowed, preventing the experiment from creating new safety gaps or unnecessary user friction at the tails. Before deploying the generated configuration, validate that the traffic allocation percentages match your feature flag system's capabilities, that the logging schema fields are actually available in your observability pipeline, and that the kill switch condition can be evaluated automatically from your real-time metrics. Run the configuration through a dry-run validation step that simulates requests across the full risk score range and confirms that routing behaves as expected at boundary values.

This prompt is designed to be called by a safety routing service at configuration time, not at request time. The service should invoke it whenever an operator or CI/CD pipeline triggers an A/B test configuration change. The prompt expects a structured [EXPERIMENT_SPEC] containing the experiment name, traffic split percentages, control and treatment threshold values, harm categories in scope, and the duration or request cap. The output is a machine-readable configuration object that downstream routers consume to assign each incoming request to an arm and apply the corresponding refusal threshold.

Validation and deployment flow: Before the generated configuration is pushed to production, validate it against a strict schema. Check that traffic split percentages sum to 100, that threshold values fall within the defined risk score range (e.g., 0.0–1.0), that harm category identifiers match the production taxonomy, and that the experiment name does not collide with an active experiment. If validation fails, log the failure with the raw model output and the schema violation details, then retry the prompt once with the validation errors appended to [CONSTRAINTS]. If the retry also fails, escalate to the on-call safety engineer and block the configuration push. On successful validation, write the configuration to a version-controlled experiment store and emit an audit event with the configuration hash, operator identity, and timestamp.

Runtime wiring: The safety routing service reads the active experiment configuration on each request. It hashes the request ID (or a stable session identifier) and maps the hash to an arm based on the traffic split. For the control arm, apply the baseline threshold; for the treatment arm, apply the experimental threshold. Log the assigned arm, the risk score from the safety classifier, the threshold applied, and the final routing decision (allow, warn, redirect, review, block) to your observability platform. Measurement harness: Build a dashboard that compares refusal rate, user satisfaction signals (e.g., appeal rate, thumbs-down ratio), and safety incident counts between arms. Use a chi-squared test or Bayesian A/B test framework to determine whether differences are statistically significant before concluding the experiment. Do not rely on the model to interpret experiment results—this prompt only generates the configuration, not the analysis.

What to avoid: Never allow the model to directly modify production thresholds without the validation and approval pipeline described above. Do not use this prompt for real-time per-request threshold decisions—that belongs to the Risk Threshold Gating Decision Prompt. Avoid running overlapping experiments on the same traffic segment; the configuration store should enforce mutual exclusion. If the experiment involves regulated domains, add a mandatory human approval step before the configuration is activated, and ensure the audit trail captures the approver's identity and rationale.