Cherry-Picked Data Detection Prompt

This prompt is a red-team and QA instrument designed to stress-test a verification pipeline's resilience against selective evidence. Its primary job is to identify claims that are technically true but misleading because they rely on outlier data points, truncated time ranges, or suppressed baselines. The ideal user is a verification system tester, a QA engineer, or an MLOps engineer who is responsible for probing a production fact-checking system for failure modes before deployment. You should use this prompt when you have a claim that appears to be supported by a specific data point, and you need to automatically flag whether the supporting evidence is unrepresentative of the broader dataset or context.

This prompt is not a general-purpose fact-checking tool. Do not use it to verify a claim against a knowledge base, to assess source authority, or to detect outright falsehoods. It is specifically designed for the adversarial testing scenario where the input is a claim-evidence pair that has already been extracted and matched. The prompt's value is in its ability to articulate missing context—such as an unmentioned time period, a suppressed baseline, or a conveniently ignored subgroup—that a naive verification system would miss. For example, a claim that 'profits increased 20% in Q3' is not false if it's true, but this prompt should flag it if the evidence ignores a 30% drop in Q2, revealing the cherry-pick.

Before integrating this prompt into a test suite, you must define the expected output schema and evaluation criteria. A successful detection not only flags the claim as cherry-picked but also specifies the missing context and the additional evidence fields required for a fair assessment. The next step is to pair this prompt with a set of known cherry-picked and non-cherry-picked examples to measure its precision and recall. Avoid deploying this as a user-facing feature without a human review step, as an overzealous detection model could incorrectly flag legitimate, focused analysis as cherry-picking, eroding user trust.

This template is designed to be copied directly into your verification harness. It instructs the model to act as a statistical-evidence auditor, analyzing a claim for signs of selective data presentation. The prompt uses square-bracket placeholders for all dynamic inputs, allowing you to swap in different claims, evidence contexts, output schemas, and risk-level constraints without rewriting the core instruction logic. The template is structured to force the model to identify missing baselines, truncated time ranges, outlier reliance, and suppressed comparison groups before rendering a verdict.

textCopy
You are a verification auditor specialized in detecting cherry-picked data. Your task is to analyze a factual claim and determine whether it relies on selective evidence presentation. You must not evaluate the truth of the claim itself, but rather whether the supporting data pattern is representative.

## INPUT
Claim: [CLAIM_TEXT]
Provided Evidence Context (if any): [EVIDENCE_CONTEXT]
Domain: [DOMAIN]

## OUTPUT SCHEMA
Return a single JSON object with the following fields:
{
  "cherry_picking_detected": boolean,
  "cherry_picking_patterns": [
    {
      "pattern_type": "truncated_time_range" | "suppressed_baseline" | "outlier_highlighting" | "missing_comparison_group" | "subgroup_isolation" | "other",
      "description": "string explaining the specific pattern found",
      "missing_context": "string describing what context or data is absent"
    }
  ],
  "required_evidence_for_fair_verification": ["string"],
  "confidence": "high" | "medium" | "low",
  "rationale": "string explaining the overall assessment"
}

## CONSTRAINTS
- Do not evaluate the truth of the claim itself. Only assess whether the data presentation pattern is selective.
- If no evidence context is provided, flag the claim as unverifiable and list the minimum evidence required.
- If the claim contains numerical comparisons, check for missing baselines, denominators, or timeframes.
- If the claim references a study or dataset, check for omitted subgroups, time windows, or conditions.
- For [RISK_LEVEL] = "high", require explicit source grounding before any pattern flag. For "low", flag patterns based on structural analysis alone.
- If you cannot determine whether cherry-picking occurred, set confidence to "low" and explain what information is missing.

## EXAMPLES
[EXAMPLES]

To adapt this template, replace each square-bracket placeholder with your application's runtime values. [CLAIM_TEXT] should receive the raw claim string from your extraction pipeline. [EVIDENCE_CONTEXT] can be populated with retrieved passages, source metadata, or left empty to test abstention behavior. [DOMAIN] helps the model calibrate its expectations—a medical claim requires different baseline awareness than a financial one. [RISK_LEVEL] controls the strictness of the grounding requirement; wire this to your triage router so high-risk claims demand source evidence before any pattern is flagged. [EXAMPLES] should be replaced with 2–4 few-shot demonstrations showing correct output for both cherry-picked and representative claims. If you are using this prompt in a batch verification pipeline, ensure the output JSON is validated against the schema before the next stage consumes it. For adversarial testing, inject claims with known cherry-picking patterns and verify that the model correctly identifies the pattern type rather than just returning a boolean.

This prompt is designed as a pre-screening filter in a verification pipeline, not a standalone tool. It should sit between claim extraction and evidence matching, flagging claims that exhibit cherry-picking patterns before they consume expensive retrieval and comparison resources. The prompt expects a single claim with its surrounding context and outputs a structured JSON object containing cherry-picking indicators, missing-context fields, and a recommended action. Wire it as a synchronous API call with a strict timeout (5-10 seconds for most models) and a JSON schema validator on the output side.

Integration pattern: Place this prompt after claim decomposition and before evidence retrieval. For each extracted claim, construct the [CLAIM] and [SOURCE_CONTEXT] variables from the upstream extraction step. The [DOMAIN] variable should be populated from a taxonomy lookup or user-provided category. Set [EVIDENCE_STANDARD] based on your use case—use 'strict' for regulated domains (finance, healthcare) and 'standard' for general content. The model returns a JSON object with fields: cherry_picking_detected (boolean), pattern_type (enum: truncated_time_range, suppressed_baseline, outlier_selection, false_dichotomy, none), missing_context_indicators (array of strings), required_evidence_fields (array of strings), confidence (0-1 float), and recommended_action (enum: proceed_to_verification, flag_for_review, reject_claim). Validate this schema immediately after the model response. If validation fails, retry once with the same input plus the validation error message appended as [PREVIOUS_ERROR]. If the second attempt also fails, route to a human review queue with the raw output attached.

Model choice and cost control: Use a fast, instruction-following model for this detection step—GPT-4o-mini, Claude 3.5 Haiku, or Gemini 1.5 Flash are good candidates. The prompt is classification-oriented with structured output, so larger models rarely improve accuracy enough to justify the latency and cost increase. Batch claims in groups of 10-20 per request if your pipeline supports it, but avoid batching if claim context varies significantly, as the model may confuse which context belongs to which claim. Log every detection result with the claim ID, pattern type, confidence score, and recommended action. This log becomes your audit trail for downstream verification decisions and helps tune the confidence threshold for auto-rejection versus human review. Start with a conservative threshold of 0.7 for flag_for_review and 0.9 for reject_claim, then adjust based on false-positive rates observed in your domain.

Failure modes to monitor: The most common production failure is false positives on legitimate narrow-scope claims—a claim that correctly cites a specific time period or data subset without deceptive intent. Mitigate this by always passing the full [SOURCE_CONTEXT] and not just the claim text. The second failure mode is missed cherry-picking when the baseline is implicit—the model may not flag a claim if the missing baseline isn't explicitly referenced in the context. For high-stakes domains, add a post-detection step that queries a knowledge base for standard baselines in the claim's domain before finalizing the detection result. Finally, watch for confidence score inflation on ambiguous claims; if your logs show high confidence on claims later proven to be cherry-picked, recalibrate by adding few-shot examples of ambiguous cases to the prompt or lowering your action thresholds.