Survey and Poll Result Verification Prompt

This prompt is designed for verification engineers, data analysts, and editorial fact-checkers who need to assess the methodological soundness of claims derived from surveys and polls. Use it when a public or internal report asserts a finding based on a survey, and you need to verify whether the sample size, margin of error, weighting, and question design support that claim. The ideal input is a specific, extracted claim alongside the reported methodology details from the source—such as the survey provider, field dates, sample size, and question wording. The prompt does not re-run raw survey data; it evaluates the reported methodology and flags gaps that undermine the claim's generalizability.

Do not use this prompt when you have access to the raw survey microdata and need to recalculate statistics, when the claim is purely qualitative with no survey methodology cited, or when the task is to compare two competing survey results rather than assess a single claim's foundation. This prompt is also inappropriate for verifying non-survey quantitative claims like financial metrics, A/B test results, or sensor readings—those require domain-specific numerical verification prompts with tolerance windows and unit normalization. The prompt works best when the input includes the full claim text, the reported methodology snippet, and any available context about the survey's sponsorship or fielding organization.

Before using this prompt, ensure you have extracted the claim cleanly from its source and gathered whatever methodology information is publicly available. The prompt will produce a structured methodology assessment with bias flags and a generalizability evaluation, but its output is only as reliable as the methodology details provided. If the source report omits sample size, margin of error, or question wording, the prompt will flag those gaps rather than fabricate assessments. For high-stakes verification workflows, always route outputs through human review and consider pairing this prompt with a source authority assessment to evaluate the polling organization's track record.

This prompt template is designed to be copied directly into your AI verification harness. It instructs the model to act as a survey methodologist and assess a claim derived from a poll or survey. The template uses square-bracket placeholders that you must replace with the specific claim, available methodology details, and your required output structure before execution. The prompt is self-contained and can be adapted for different models by adjusting the [OUTPUT_SCHEMA] and [CONSTRAINTS] blocks.

codeCopy
You are a survey methodologist and statistical auditor. Your task is to verify a claim derived from a survey or poll by assessing its methodological soundness, representativeness, and generalizability.

# CLAIM TO VERIFY
[CLAIM_TEXT]

# AVAILABLE METHODOLOGY DETAILS
- Sample Size: [SAMPLE_SIZE]
- Population: [TARGET_POPULATION]
- Sampling Method: [SAMPLING_METHOD]
- Mode of Data Collection: [DATA_COLLECTION_MODE]
- Margin of Error (reported): [REPORTED_MOE]
- Confidence Level (reported): [REPORTED_CONFIDENCE_LEVEL]
- Weighting Procedures: [WEIGHTING_DETAILS]
- Response Rate: [RESPONSE_RATE]
- Survey Dates: [SURVEY_DATES]
- Question Wording: [QUESTION_WORDING]
- Sponsor/Funder: [SPONSOR_INFORMATION]
- Additional Context: [ADDITIONAL_CONTEXT]

# VERIFICATION TASKS
1. Assess sample size adequacy for the target population and reported margin of error.
2. Evaluate the sampling method for potential selection bias and representativeness.
3. Check the reported margin of error calculation against the sample size and confidence level.
4. Identify non-response bias risks based on the response rate and mode of data collection.
5. Analyze the question wording for leading, ambiguous, or loaded phrasing.
6. Evaluate the weighting methodology for appropriateness and potential over-correction.
7. Assess the generalizability of the claim to the stated population.
8. Flag any undisclosed limitations that could materially affect the claim's validity.

# CONSTRAINTS
- Do not invent missing methodology details. If critical information is absent, flag it as "UNKNOWN" and explain why it matters.
- Distinguish between "the claim is mathematically inconsistent with the reported methodology" and "the methodology is insufficient to support the claim."
- Use precise statistical language. Do not round intermediate calculations.
- If the claim involves subgroup analysis, check whether the subgroup sample size is reported and sufficient.
- Flag any evidence of p-hacking, multiple comparison issues, or post-hoc storytelling.

# OUTPUT SCHEMA
[OUTPUT_SCHEMA]

# RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace each [PLACEHOLDER] with the actual data available for the claim under review. If a methodology detail is missing from the source material, set the placeholder value to "Not Provided" rather than deleting the field—this allows the model to flag the absence as a limitation. The [OUTPUT_SCHEMA] placeholder should contain your desired JSON schema, field definitions, or structured format instructions. The [RISK_LEVEL] placeholder should specify the verification rigor required: use "LOW" for internal exploratory checks, "MEDIUM" for published content with editorial review, and "HIGH" for regulated, compliance, or public-interest claims that require human sign-off before release.

Before deploying this prompt in production, pair it with a validation layer that checks the output conforms to your [OUTPUT_SCHEMA]. For high-risk claims, route outputs scoring above a confidence threshold for human review and log every verification result with the input claim, methodology details, and model version for auditability. Common failure modes include the model hallucinating missing methodology details, accepting the reported margin of error without recalculation, and failing to flag question-wording bias when the wording appears neutral but contains subtle framing. Test against a golden set of known survey flaws—such as small samples, non-probability sampling, and leading questions—to calibrate your eval thresholds before production use.

The Survey and Poll Result Verification Prompt is designed to operate as a structured verification step within a larger claims-checking pipeline. It expects a pre-extracted claim about a survey or poll result, along with the source methodology document or metadata. The prompt is not a standalone chatbot; it should be called programmatically with a strict JSON output schema so that downstream systems can parse the methodology assessment, bias flags, and generalizability evaluation without manual interpretation.

Input assembly requires three components: the [CLAIM] text (the survey-derived assertion being checked), the [METHODOLOGY_DOCUMENT] (the full methodology section, crosstabs, or technical appendix from the survey source), and an optional [REFERENCE_STANDARDS] block containing accepted industry guidelines such as AAPOR, ESOMAR, or ISO 20252 criteria. If the methodology document is retrieved via RAG, log the retrieval source, chunk IDs, and retrieval timestamp alongside the prompt call. For high-stakes verification, include the [RISK_LEVEL] parameter set to high to trigger stricter tolerance windows and mandatory human review routing.

Output validation must enforce the expected JSON schema before the result enters any downstream system. Key checks include: (1) sample_size_assessment contains a numeric reported_n and a boolean adequate_for_claimed_moe; (2) bias_flags is a non-null array with each flag having a type, severity, and evidence field; (3) generalizability_score is an integer between 1 and 5; (4) margin_of_error_verified is a boolean with an accompanying calculated_moe and claimed_moe for comparison. If the model returns malformed JSON, use a repair prompt with the schema and the raw output, but limit repair attempts to two retries before escalating to a human reviewer with the original claim and methodology attached.

Model choice and latency budgeting matter here. For high-throughput batch verification of low-risk survey claims (e.g., internal tracking polls), a fast model like GPT-4o-mini or Claude Haiku with a 5-second timeout is sufficient. For high-stakes claims (published election polls, medical survey results, regulatory submissions), route to a more capable model like GPT-4o or Claude Sonnet with a 30-second timeout and enable thinking tokens for complex methodology analysis. Always log the model version, prompt template version, and latency for audit trails.

Human review routing should be triggered automatically when any of these conditions are met: generalizability_score is 2 or lower, any bias_flags entry has severity of critical or high, margin_of_error_verified is false, or the sample_size_assessment.adequate_for_claimed_moe is false while the claim is published externally. Package the full prompt input, raw model output, parsed JSON, and any validation errors into a review queue payload. Do not auto-publish or auto-accept survey verification results that fail these gates.

Testing and eval require a golden dataset of 20–30 survey claims with known methodology flaws and correct assessments. Measure precision and recall on bias flag detection, exact match on margin_of_error_verified, and mean absolute error on calculated_moe versus ground truth. Run this eval suite on every prompt template change and log results before promoting to production. Common failure modes include the model accepting a claimed margin of error without recalculating it from sample size, missing non-response bias when response rates are buried in footnotes, and over-generalizing findings when the sample frame is clearly restricted.