Statistical Significance Check Prompt Template

This prompt is for engineering leads and AI platform teams who need to determine whether observed differences between two prompt variants are statistically meaningful or just noise. Use it after collecting evaluation scores from an A/B test where Variant A and Variant B were run against a shared set of inputs. The prompt expects raw per-sample scores, not pre-aggregated summaries, so it can compute proper test statistics. This is a critical gate in the eval harness stage, positioned after metric collection and before making a ship or rollback decision.

The ideal workflow requires you to provide paired or unpaired raw scores from your evaluation runs. For example, if you ran both prompts against the same 100 test cases, supply the 100 scores for Variant A and the 100 scores for Variant B. The prompt will then select the appropriate test (e.g., paired t-test, Wilcoxon signed-rank, or Mann-Whitney U) based on the data characteristics you describe, compute the p-value, estimate effect size, and flag whether your sample size is adequate to detect the observed difference. It will also warn you if you are running multiple comparisons without correction.

Do not use this prompt when you only have aggregate metrics like means and standard deviations without the underlying per-sample data. It is also inappropriate when the evaluation metric is purely categorical without a defined ordering, or when the two variants were not tested on the same or randomly equivalent inputs. In those cases, you risk the prompt producing misleading statistics from insufficient data. After receiving the output, always review the test assumptions the prompt states and verify they match your experimental design before acting on the recommendation.

This template is designed to be dropped directly into your eval harness after you have collected per-sample scores from two prompt variants. It expects raw score arrays and a set of analysis parameters, and it returns a structured statistical report. The prompt enforces a strict output schema so that downstream automation—such as release gates or dashboards—can consume the results without manual parsing. Use square-bracket placeholders to inject your experiment data and configuration before each run.

textCopy
You are a statistical analyst evaluating an A/B test between two prompt variants.
Your task is to determine whether the observed difference in scores is statistically significant.

## Input Data
- Scores for Variant A: [SCORES_A]
- Scores for Variant B: [SCORES_B]
- Alpha (significance threshold): [ALPHA]
- Test type: [TEST_TYPE]  // Options: "two-sided", "one-sided-greater", "one-sided-less"
- Correction method for multiple comparisons: [CORRECTION_METHOD]  // Options: "none", "bonferroni", "holm", "benjamini-hochberg"
- Minimum effect size of practical interest: [MIN_EFFECT_SIZE]
- Sample size adequacy check: [CHECK_SAMPLE_SIZE]  // Options: "yes", "no"

## Output Schema
Return a JSON object with the following structure:
{
  "test_parameters": {
    "alpha": number,
    "test_type": string,
    "correction_method": string,
    "min_effect_size": number
  },
  "descriptive_statistics": {
    "variant_a": { "n": number, "mean": number, "std": number, "median": number },
    "variant_b": { "n": number, "mean": number, "std": number, "median": number }
  },
  "test_results": {
    "test_used": string,
    "test_statistic": number,
    "p_value_raw": number,
    "p_value_corrected": number,
    "is_significant": boolean,
    "confidence_interval_95": [number, number],
    "effect_size_cohens_d": number,
    "effect_size_interpretation": string
  },
  "sample_size_assessment": {
    "is_adequate": boolean,
    "observed_power": number,
    "recommendation": string
  },
  "multiple_comparison_note": string,
  "overall_conclusion": string,
  "caveats": [string]
}

## Constraints
- Use Welch's t-test by default unless the score distributions are known to have equal variance.
- If the data appears non-normal (e.g., highly skewed), note this in caveats and consider recommending a Mann-Whitney U test instead.
- If [CHECK_SAMPLE_SIZE] is "yes", compute observed power and flag inadequate samples.
- If [CORRECTION_METHOD] is not "none", apply the specified correction and report both raw and corrected p-values.
- Interpret Cohen's d using standard thresholds: 0.2 small, 0.5 medium, 0.8 large.
- The overall_conclusion must state clearly whether the result is statistically significant and practically meaningful.
- Do not fabricate data. If inputs are insufficient, return an error object with a "error" key and a message.

## Risk Level
[HIGH] — Statistical misinterpretation can lead to shipping regressions. Human review of the conclusion is required before acting on the result.

After pasting this template into your harness, replace each placeholder with live data from your experiment. The [SCORES_A] and [SCORES_B] arrays should contain per-example evaluation scores from your LLM judge or human raters. Set [ALPHA] to your significance threshold (commonly 0.05). Choose [TEST_TYPE] based on whether you are testing for any difference or a directional improvement. If you are running multiple pairwise comparisons across several prompt variants, set [CORRECTION_METHOD] to a non-"none" value to avoid inflated false-positive rates. The [MIN_EFFECT_SIZE] field prevents you from shipping a statistically significant but practically meaningless change—set it to the smallest Cohen's d you would consider worth the migration cost. Always route the final overall_conclusion and caveats to a human reviewer before gating a release on this result.

The Statistical Significance Check prompt is designed to be called programmatically within an A/B test analysis pipeline, not as a one-off chat interaction. The typical integration point is after raw metric data has been collected from two or more prompt variants. Your application layer should first compute the descriptive statistics (sample sizes, means, variances) and then pass those structured inputs into the prompt template. This separation ensures the LLM focuses on interpretation and statistical reasoning rather than raw arithmetic, which is error-prone in language models. The prompt expects a JSON input block containing variant labels, sample sizes, means, standard deviations, and the chosen alpha level. Your harness should validate this input shape before calling the model.

Implement a strict validation layer around both the input and output of this prompt. Before calling the model, confirm that all required fields are present, sample sizes are positive integers, standard deviations are non-negative, and alpha is between 0 and 1. After receiving the model response, parse the output against the expected schema: a JSON object containing significant (boolean), p_value (float), effect_size (float with Cohen's d or equivalent), sample_size_adequate (boolean), multiple_comparison_warning (boolean), and interpretation (string). If the model returns malformed JSON, implement a single retry with a repair prompt that includes the raw output and the schema. If the retry also fails, escalate to a human reviewer rather than silently falling back. Log every call with the input parameters, raw model output, parsed result, and retry count for auditability.

For high-stakes decisions—such as promoting a prompt variant to production or concluding a multi-week experiment—require human approval in the loop. The harness should flag any result where p_value is between 0.01 and 0.05, where sample_size_adequate is false, or where multiple_comparison_warning is true. These flagged results should be routed to a review queue with the full context: the input statistics, the model's interpretation, and a link to the raw trace. Do not auto-apply decisions based on the model's significant boolean alone. Model choice matters here: prefer models with strong quantitative reasoning (such as Claude 3.5 Sonnet or GPT-4o) and set temperature to 0 for deterministic outputs. Avoid using this prompt with small or older models that are known to hallucinate statistical calculations.

When embedding this prompt in a CI/CD pipeline for automated prompt regression gates, cache results aggressively. If the same input statistics are evaluated multiple times across pipeline runs, reuse the cached interpretation to avoid unnecessary API calls and non-determinism. Store the cache key as a hash of the normalized input JSON. Additionally, run a periodic calibration check by feeding the prompt a set of hand-verified statistical scenarios with known outcomes (e.g., a clearly significant result with large effect size, a clearly underpowered comparison). If the model's outputs drift from expected interpretations across calibration runs, trigger an alert and pause automated gating until the eval harness is reviewed. This prevents silent degradation in statistical judgment from model updates or provider-side changes.