Statistical Significance Misrepresentation Detection Prompt

This prompt is designed for verification engineers and adversarial testing teams who need to automatically flag claims that misuse or misrepresent statistical significance. The core job-to-be-done is detecting when a quantitative claim reports a p-value without necessary context, confuses statistical significance with practical importance, cherry-picks significance thresholds, or implies causation from correlation under the cover of 'significance.' The ideal user is someone integrating this into a pre-deployment adversarial test suite or a production claim-verification pipeline where numerical claims must be stress-tested before they reach end users or automated fact-checking systems.

Use this prompt when you have extracted a discrete numerical claim from source content and need to assess its statistical rigor before passing it to downstream verification or publication. Required context includes the full claim text, the reported statistical measures (p-values, confidence intervals, effect sizes, sample sizes), the domain of the claim (medical, social science, A/B testing, etc.), and any source metadata that reveals study design or data collection constraints. The prompt expects a structured output with specific flags for common misrepresentation patterns—missing effect size, p-hacking indicators, threshold cherry-picking, and causal overreach—along with required disclosure fields that a rigorous claim should include.

Do not use this prompt for general fact-checking of non-quantitative claims, for verifying the raw data behind a statistical test, or as a replacement for a trained statistician's review. It is a detection and flagging tool, not a statistical calculator. It will not recompute p-values or confirm the correctness of reported numbers. In regulated domains such as pharmaceutical claims, clinical trial reporting, or financial risk disclosures, always route flagged outputs for human expert review. The prompt is most effective as part of a layered verification architecture where it serves as a statistical-rigor triage step before claims reach evidence-matching, contradiction detection, or audit-trail generation stages. Pair it with the Confidence Score Calibration Under Adversarial Input prompt to ensure the flagging thresholds remain stable under edge-case probing.

This template is designed to be dropped into a verification agent or batch processing pipeline. It instructs the model to act as a statistical rigor auditor, examining a single claim for common misrepresentations of statistical significance. The prompt uses square-bracket placeholders for all dynamic inputs, making it safe for template engines and configuration management systems. Before deploying, you must define your evidence sources, output schema, and risk thresholds.

textCopy
You are a statistical rigor auditor. Your task is to examine a claim and flag any misrepresentation of statistical significance.

## INPUT
Claim: [CLAIM_TEXT]
Context: [SURROUNDING_PARAGRAPH]
Evidence: [RETRIEVED_EVIDENCE_JSON]

## CONSTRAINTS
- Do not evaluate the truth of the claim itself. Focus only on how statistical significance is presented.
- If the claim reports a p-value without effect size, confidence interval, or sample size, flag it as incomplete.
- Distinguish between statistical significance and practical significance. A significant p-value does not mean a large or important effect.
- Flag any language that implies causation from a correlational or significant result without further evidence.
- Check for p-hacking indicators: rounded p-values just below 0.05, selective reporting of outcomes, or undisclosed multiple comparisons.
- If the claim uses "significant" without a statistical context, note the ambiguity.

## OUTPUT_SCHEMA
Return a single JSON object with the following fields:
{
  "claim_id": "string",
  "flags": [
    {
      "flag_type": "MISSING_EFFECT_SIZE | CONFUSING_PRACTICAL_SIGNIFICANCE | CAUSAL_OVEREACH | P_HACKING_INDICATOR | AMBIGUOUS_LANGUAGE | NO_ISSUE",
      "severity": "CRITICAL | WARNING | INFO",
      "description": "string",
      "required_disclosure": "string"
    }
  ],
  "overall_rigor_score": "HIGH | MEDIUM | LOW | UNVERIFIABLE",
  "human_review_required": true/false
}

## EXAMPLES
Claim: "The new drug showed a significant improvement (p<0.05)."
Output: { "flags": [{ "flag_type": "MISSING_EFFECT_SIZE", "severity": "CRITICAL", "description": "Reports p-value without effect size or confidence interval.", "required_disclosure": "Report the effect size (e.g., Cohen's d) and 95% confidence interval." }], "overall_rigor_score": "LOW", "human_review_required": true }

Claim: "We observed a statistically significant 2% increase in click-through rate (p=0.03, 95% CI [0.5%, 3.5%])."
Output: { "flags": [{ "flag_type": "NO_ISSUE", "severity": "INFO", "description": "Reports p-value, effect size, and confidence interval. No misrepresentation detected.", "required_disclosure": "None" }], "overall_rigor_score": "HIGH", "human_review_required": false }

## RISK_LEVEL
[RISK_LEVEL]

## INSTRUCTIONS
Analyze the claim against the provided evidence and context. Output only the JSON object. Do not include any other text.

To adapt this template, start by replacing the [CLAIM_TEXT] and [SURROUNDING_PARAGRAPH] placeholders with data from your claim extraction pipeline. The [RETRIEVED_EVIDENCE_JSON] placeholder should receive structured evidence from your retrieval system, including source metadata, publication dates, and any numerical data needed to verify the claim's statistical reporting. The [RISK_LEVEL] placeholder allows you to inject domain-specific policy instructions—for example, a medical application might require mandatory human review for any CRITICAL flag, while a low-stakes marketing analysis might only escalate on CAUSAL_OVEREACH. After adapting, run this prompt against your golden dataset of known statistical misrepresentations to calibrate the overall_rigor_score thresholds and flag severity mappings before production deployment.

This prompt is designed to operate as a specialized verification microservice within a larger fact-checking pipeline. It should be deployed after claim extraction and before evidence matching, specifically filtering for claims that contain numerical or statistical language. The implementation harness must ensure the prompt receives clean, well-formed claims and that its output is validated before downstream systems consume the statistical-rigor flags. Because this prompt deals with nuanced statistical interpretation, it is particularly sensitive to input quality—garbage claims will produce garbage flags. The harness should include a pre-filter that routes only claims containing statistical signal words (p-value, significant, correlation, confidence interval, effect size, sample size, etc.) to this prompt, reducing unnecessary inference costs on non-statistical claims.

Integration pattern: Wire this prompt as a synchronous validation step with a strict timeout (5-10 seconds for most models) and a JSON schema validator on the output. The recommended flow: (1) Claim extractor outputs atomic claims → (2) Statistical keyword router sends matching claims here → (3) This prompt produces a structured JSON response with flags, required_disclosures, and rigor_score → (4) A post-processing validator checks that all required fields are present, enum values are valid, and the output conforms to the expected schema → (5) If validation fails, retry once with the same input and a stronger format reminder appended to the prompt → (6) If validation fails again, log the failure and route the claim to a human review queue with the raw model output attached. For high-throughput pipelines, batch claims in groups of 5-10 to amortize prompt overhead, but keep batches small enough that individual claim context isn't diluted.

Model selection and cost considerations: This prompt requires strong reasoning capabilities and statistical literacy. Use a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) rather than a lightweight classifier. The prompt's output is high-stakes—false negatives on statistical misrepresentation can propagate misleading claims through the verification pipeline. Set temperature=0 for deterministic outputs and use structured output mode (JSON mode or function calling) to enforce the schema. Log every invocation with: input claim text, model response, validation pass/fail, retry count, and final disposition (passed, flagged, errored). This audit trail is essential for debugging false-positive flag patterns and for governance reviews of the verification pipeline. If using a model router, escalate to a stronger model when the primary model returns low-confidence or malformed outputs rather than silently accepting degraded results.

Human review integration: Claims that receive a rigor_score below 0.5 or that trigger critical severity flags should be routed to a human review queue with the full prompt output and the original source context. The review interface should display: the original claim, the model's flag list, the required disclosure fields, and a simple approve/override/reject action. Track reviewer decisions to build a calibration dataset for future prompt improvements. Avoid fully automated rejection of claims based solely on this prompt's output—statistical misrepresentation detection involves judgment calls that benefit from human oversight, especially in regulated domains like clinical research or financial reporting. The prompt's value is in surfacing potential issues efficiently, not in making final determinations.