Statistical Significance and P-Value Verification Prompt

This prompt is designed for analysts, verification engineers, and data teams who need to validate statistical significance claims found in reports, papers, marketing materials, or internal analyses. Use it when you have a claim that includes a p-value, a significance statement, or a hypothesis test conclusion and you need to verify whether the test was appropriate, the p-value is correctly reported, effect sizes are disclosed, and multiple comparison corrections were applied where needed. The primary job-to-be-done is methodological auditing of a stated claim, not re-running the original analysis against raw data. You provide the claim text, the reported test details, and any available context about the study design, and the prompt returns a structured assessment of reporting completeness, test appropriateness, and common failure patterns such as p-hacking or significance threshold shopping.

This prompt belongs in verification pipelines that process quantitative claims after content or analysis exists. It is not a replacement for running the actual statistical test against raw data; it validates the reporting and methodological soundness of the claim as stated. The prompt assumes you can provide the claim text, the reported test details (test type, sample size, p-value, effect size if available), and any available context about the study design. For regulated or high-stakes domains such as clinical trials or drug approvals, always route outputs to human review before any decision. The prompt is most effective when paired with a structured output schema that forces the model to separate factual reporting checks (e.g., 'is the p-value within valid range?') from judgment calls (e.g., 'is the test appropriate for this data structure?').

Do not use this prompt when you need to compute a p-value from raw data, when the claim involves Bayesian methods without frequentist reporting, or when the statistical test is so domain-specific that general verification heuristics do not apply. The prompt is also insufficient for detecting sophisticated fraud such as fabricated data; it checks reporting consistency and methodological soundness, not data integrity. Before deploying, define your evaluation criteria: what counts as a pass for test appropriateness, what severity levels you assign to missing effect sizes versus missing corrections, and at what confidence threshold you route to human review. Start with a small set of known-true and known-false claims to calibrate the prompt's behavior before scaling to production volumes.

This prompt template is designed to be pasted directly into your verification workflow. It instructs the model to act as a statistical auditor, checking a provided claim for test appropriateness, p-value accuracy, effect size reporting, and common pitfalls like p-hacking or significance threshold shopping. Replace the square-bracket placeholders with the specific claim text, the available source data or methodology description, and your required output structure before sending it to the model.

textCopy
You are a statistical verification auditor. Your task is to evaluate a statistical significance claim against provided context. You must be precise, skeptical, and flag all assumptions.

## CLAIM TO VERIFY
[CLAIM_TEXT]

## AVAILABLE CONTEXT
[CONTEXT]

## OUTPUT SCHEMA
Return a single JSON object with the following fields:
- claim_summary: A one-sentence restatement of the statistical claim being made.
- test_identified: The name of the hypothesis test or statistical method used (e.g., 'two-tailed t-test', 'chi-squared test of independence').
- test_appropriateness: An assessment ('APPROPRIATE', 'INAPPROPRIATE', 'UNCLEAR') with a brief justification.
- p_value_reported: The p-value stated in the claim, or null if not provided.
- p_value_verifiable: A boolean indicating if the p-value can be recalculated from the context.
- effect_size_reported: A boolean indicating if an effect size (e.g., Cohen's d, odds ratio) is reported alongside the p-value.
- multiple_comparison_correction: An assessment ('NOTED', 'NOT_MENTIONED', 'NOT_APPLICABLE') indicating if corrections for multiple comparisons were addressed.
- p_hacking_indicators: A list of strings describing any red flags for p-hacking, such as 'no pre-registered primary outcome', 'multiple uncorrected tests', or 'flexible exclusion criteria'.
- significance_threshold: The alpha level used (e.g., 0.05), or null if not stated.
- threshold_shopping_risk: An assessment ('LOW', 'MEDIUM', 'HIGH') of the risk that the significance threshold was chosen after seeing the data.
- overall_verdict: A final assessment ('SUPPORTED', 'UNSUPPORTED', 'PARTIALLY_SUPPORTED', 'INSUFFICIENT_CONTEXT').
- verdict_rationale: A concise explanation of the overall verdict, referencing specific checks.
- requires_human_review: A boolean indicating if the claim involves high-stakes decisions or complex methodology requiring expert review.

## CONSTRAINTS
- Do not invent data. If the context is insufficient to verify a calculation, state that clearly.
- Distinguish between 'statistical significance' and 'practical significance' in your rationale.
- If the claim uses causal language (e.g., 'leads to', 'increases'), flag it unless the study design supports causal inference.
- If the p-value is reported as 'p < 0.05' without an exact value, note this as a precision limitation.

To adapt this template, start by pasting the exact text of the claim into [CLAIM_TEXT]. For [CONTEXT], provide the methodology section of the study, a data summary table, or the full dataset if available. The more context you provide, the more the model can verify calculations directly rather than just assessing reporting quality. If you are integrating this into an automated pipeline, parse the JSON output and use the requires_human_review boolean to route high-risk or complex claims to a subject-matter expert. For high-stakes domains like clinical trials or financial risk models, always set a policy that a human must review any verdict that is not SUPPORTED.

This prompt is designed to operate as a verification microservice within a larger fact-checking pipeline. It expects a structured input payload containing the claim text, the reported statistical test details, and the source data or summary statistics. The output is a structured verification result that downstream systems can consume directly. Do not use this prompt in isolation for high-stakes decisions—always pair it with programmatic validation of the output schema, tolerance checks on reported values, and a human review gate when the prompt flags potential p-hacking, threshold shopping, or test appropriateness violations.

Wiring the prompt into an application requires a pre-processing layer that normalizes inputs before the model call. Extract the claim from upstream claim decomposition, parse the reported test type (t-test, chi-square, ANOVA, etc.), reported p-value, sample size, and effect size if available. Package these into the [CLAIM_DETAILS] placeholder as structured JSON. For the [SOURCE_DATA] placeholder, provide either raw data arrays or summary statistics (means, standard deviations, counts) depending on what's available. The [CONSTRAINTS] placeholder should include your organization's significance threshold (e.g., alpha=0.05), correction method expectations (Bonferroni, Benjamini-Hochberg), and minimum effect size reporting requirements. After the model returns its verification result, run a schema validator to confirm the output matches the expected JSON structure with required fields: test_appropriateness, p_value_accuracy, effect_size_reported, correction_applied, p_hacking_indicators, overall_verdict, and confidence_score. Reject and retry any response that fails schema validation, with a maximum of two retries before escalating to human review.

Model choice and tool integration matter here. Use a model with strong quantitative reasoning capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). For the [TOOLS] placeholder, you can optionally provide a calculator function if your harness supports tool-use loops—this allows the model to recompute test statistics from raw data rather than relying solely on internal reasoning. If you implement tool-use, log every tool call and its result for auditability. Retry logic should distinguish between schema failures (retry with the same prompt plus the validation error message) and low-confidence outputs (retry with explicit instruction to flag uncertainties rather than guess). For RAG integration, if your pipeline retrieves methodological guidelines or domain-specific significance conventions, inject them into the [CONTEXT] placeholder before the model call. This is particularly important for fields like clinical trials where significance thresholds and correction methods are regulated.

Production hardening requires attention to several failure modes. First, the prompt can hallucinate p-value recalculations when given incomplete summary statistics—always cross-validate any model-computed p-value against a deterministic statistical library in your application code. Second, the prompt may over-flag or under-flag p-hacking indicators depending on how strictly you define the pattern taxonomy in [CONSTRAINTS]; calibrate this with a labeled test set of known p-hacking cases. Third, when the source data contains multiple comparisons without explicit correction, the model may inconsistently detect this—add a programmatic check that counts the number of hypothesis tests in the input and flags any claim where correction_applied is false but test count exceeds 1. Human review routing should trigger when: (a) the model's confidence score falls below your threshold (suggest 0.7 for initial deployment), (b) p-hacking indicators are detected, (c) the test type is flagged as inappropriate, or (d) the schema validator fails after retry exhaustion. Package the full prompt input, model output, validation results, and any tool-call logs into a review packet for the human analyst.

What to avoid: Do not treat this prompt as a replacement for a statistical computing environment. It verifies claims, it does not perform original analysis. Do not skip the programmatic p-value cross-check—model arithmetic errors on statistical calculations are well-documented. Do not deploy without a labeled evaluation set covering common failure patterns: p-hacking (data dredging, optional stopping, outcome switching), threshold shopping (reporting p<0.05 when p=0.049 vs p=0.051), test assumption violations (using parametric tests on non-normal data), and multiple comparison neglect. Run this eval set on every prompt version change and gate deployment on maintaining or improving F1 scores for each failure category.