Invented Statistic Flagging Prompt for Report Generation

This prompt is designed for business intelligence and automated reporting pipelines where a language model has already generated a draft report from source data. Its job is to act as a post-generation validation layer: it scans the generated report text, identifies every quantitative claim, percentage, or statistical figure, and cross-references each against the provided source data. Any statistic that cannot be traced to the source input is flagged with a severity level and a recommended action. Use this prompt when your product cannot afford fabricated numbers reaching dashboards, client deliverables, or regulatory filings. It assumes you already have a generated report and the original source data available. It does not rewrite the report; it produces a structured audit artifact that your application can use to block, quarantine, or annotate the output before it reaches users.

The ideal user is an engineering lead or platform developer integrating this as a guardrail in an automated report generation system. The prompt requires two concrete inputs: the [GENERATED_REPORT] text and the [SOURCE_DATA] from which the report should have been derived. The source data should be structured (JSON, CSV, or a well-defined table format) so the model can perform precise value matching. Do not use this prompt when the source data is ambiguous, when the report is expected to contain analyst interpretation that goes beyond the data, or when you need real-time streaming validation—this is a batch audit step. It is also not a replacement for a deterministic numerical validation script; use it when the report is in natural language and the mapping between claims and source fields requires semantic understanding.

Before wiring this into production, define what happens after the audit artifact is produced. A common pattern is to set a threshold: if any statistic is flagged with severity HIGH, block the report from publication and route it for human review. For MEDIUM flags, annotate the report with a disclaimer. For LOW flags, log the finding and allow publication. You should also implement a retry loop: if the audit prompt itself fails to produce valid JSON, retry with a stricter output schema instruction before escalating. Finally, periodically sample audit artifacts and manually verify that the prompt is not producing false positives (flagging legitimate statistics) or false negatives (missing fabricated numbers), as both failure modes erode trust in the validation layer.

This prompt template is designed to be used as a post-generation validation step in your report pipeline. After a model produces a report draft, you pass that draft into this prompt along with the source data that was used to generate it. The prompt instructs the model to act as a strict auditor, cross-referencing every quantitative claim, percentage, and statistical figure against the provided source material. The output is a structured audit report that flags each invented statistic with a severity level and a recommended action, enabling your application to either strip the fabricated figures or escalate for human review before the report reaches an end user.

textCopy
You are a statistical fact-checking auditor for automated report generation. Your task is to compare a generated report against the source data that was provided to the model. You will identify every quantitative claim, percentage, statistical figure, or numeric assertion in the report that cannot be traced to the source data.

## INPUT

[REPORT_TEXT]

[SOURCE_DATA]

## INSTRUCTIONS

1. Extract every discrete quantitative claim from the report. This includes:
   - Percentages and proportions
   - Absolute numbers, counts, and totals
   - Rates, ratios, and averages
   - Monetary amounts and financial figures
   - Date-based comparisons and growth figures
   - Rankings and ordinal claims with numeric backing
2. For each extracted claim, search the [SOURCE_DATA] for evidence that directly supports the figure. A claim is grounded only if the exact number or a calculation that yields that number is present in the source data.
3. Classify each claim as:
   - **GROUNDED**: The exact figure or a directly calculable equivalent exists in the source data.
   - **INVENTED**: No supporting evidence exists in the source data. The model fabricated the number.
   - **DISTORTED**: The source data contains a related figure, but the report's number is materially different and unsupported.
4. For each INVENTED or DISTORTED claim, assign a severity level:
   - **CRITICAL**: The invented figure could cause financial, legal, or safety harm if relied upon.
   - **HIGH**: The figure is central to the report's conclusions or recommendations.
   - **MEDIUM**: The figure is supporting detail but not central to the main argument.
   - **LOW**: The figure is incidental or decorative.
5. For each INVENTED or DISTORTED claim, recommend one of:
   - **REMOVE**: Strip the claim entirely from the report.
   - **REPLACE**: Suggest a grounded figure from the source data if a related one exists.
   - **HUMAN_REVIEW**: Escalate for manual verification when the claim is critical but source ambiguity exists.

## OUTPUT FORMAT

Return a JSON object with the following structure:

{
  "audit_summary": {
    "total_claims_found": <integer>,
    "grounded_claims": <integer>,
    "invented_claims": <integer>,
    "distorted_claims": <integer>,
    "critical_findings": <integer>,
    "high_findings": <integer>,
    "overall_assessment": "PASS" | "FLAGGED" | "FAIL"
  },
  "findings": [
    {
      "claim_id": <string>,
      "claim_text": <string>,
      "claim_location": <string>,
      "classification": "GROUNDED" | "INVENTED" | "DISTORTED",
      "severity": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
      "source_evidence": <string or null>,
      "recommended_action": "KEEP" | "REMOVE" | "REPLACE" | "HUMAN_REVIEW",
      "replacement_suggestion": <string or null>,
      "reasoning": <string>
    }
  ]
}

## CONSTRAINTS

- Do not flag claims that are supported by the source data, even if they are paraphrased or reformatted.
- If the source data is insufficient to verify a claim, classify it as INVENTED rather than assuming it is correct.
- Do not invent source evidence. If you cannot find it, mark the claim as INVENTED.
- For DISTORTED claims, include the correct source figure in the replacement_suggestion field.
- If no quantitative claims exist in the report, return an empty findings array with an overall_assessment of "PASS".
- Preserve the exact claim_text as it appears in the report for traceability.

To adapt this template for your pipeline, replace [REPORT_TEXT] with the full generated report and [SOURCE_DATA] with the raw data, retrieved context, or structured records that were provided to the generation model. If your source data is large, consider chunking it or providing only the sections relevant to the report's domain. The output JSON can be parsed by your application to automatically strip claims marked REMOVE, substitute REPLACE suggestions, or route CRITICAL and HIGH findings to a human review queue. For high-stakes domains like financial reporting or clinical summaries, always require human sign-off on any finding above MEDIUM severity before the report is published.

This prompt is designed to sit as a post-generation validation step in a report generation pipeline. After a language model produces a draft report, the output is routed to this prompt along with the original source documents or data extracts that were provided as context. The prompt's job is to identify quantitative claims—percentages, dollar amounts, growth rates, survey results, and other statistics—that appear in the generated text but cannot be traced back to the source material. The output is a structured flagging report that downstream systems can use to decide whether to strip unsupported figures, request a regeneration, or escalate for human review.

The implementation harness should enforce a strict contract on both inputs and outputs. On the input side, the system must provide the generated report text and a structured array of source excerpts with identifiers. Each source excerpt should include a source_id, the raw text, and optionally a section_label to help the model produce traceable flagging. On the output side, the harness should validate that the model returns a JSON array where each flag object contains the required fields: flagged_statistic (the exact text from the report), statistic_type (e.g., percentage, absolute_number, ratio, trend_claim), severity (high, medium, low), source_grounding_status (unsupported, partially_supported, contradicted), nearest_source_evidence (the closest matching source excerpt or null), and recommended_action (remove, replace_with_sourced, flag_for_review). A JSON schema validator should reject any response that does not conform to this structure and trigger a retry with the validation error included in the retry prompt.

For production deployment, implement a retry loop with a maximum of two attempts. If the first call returns invalid JSON or missing required fields, append the validation error to a retry instruction and resubmit. If the second attempt also fails, log the failure, attach the raw output to the report metadata, and route the entire report to a human review queue with a validation_failed status. For high-severity flags—statistics marked as unsupported with high severity—the system should automatically strip those figures from the report before publication and insert a [DATA PENDING VERIFICATION] placeholder. Medium-severity flags can be surfaced as annotations in the report UI for reviewer attention without blocking publication. Low-severity flags, such as rounding discrepancies or minor formatting issues, can be logged for trend analysis without interrupting the workflow. All flagging events should be written to an observability store with the report ID, model version, prompt version, and timestamp for auditability and prompt performance tracking over time.