Stakeholder-Friendly Hallucination Report Prompt Template

This prompt is designed for quality engineers and AI program managers who need to translate raw hallucination trace data into a structured, business-focused report for product and content stakeholders. The goal is to communicate unsupported claim frequency, grounding failure categories, and affected content domains without requiring the audience to understand retrieval-augmented generation internals or trace schemas. Use this prompt when you have already classified hallucinations from production traces and need a decision-ready summary that frames impact, trends, and recommended actions.

The ideal user has completed hallucination classification upstream—meaning each trace event has already been labeled with a failure category (e.g., fabricated entity, unsupported statistic, misattributed source) and mapped to a content domain. The prompt expects structured input: a dataset of classified hallucination events with timestamps, severity scores, affected document sections, and retrieval context summaries. Without this pre-classification, the prompt will produce unreliable abstractions that hide diagnostic signal rather than surface it. You must also provide a [CONSTRAINTS] block specifying the audience's technical depth, preferred format, and any business metrics that should anchor the impact framing.

Do not use this prompt for real-time alerting, raw trace debugging, or technical root-cause analysis intended for an engineering audience. It is not a substitute for trace-level diagnosis tools, nor should it be used when the underlying hallucination classifications have not been validated. If your hallucination classifier has a known false-positive rate above your risk tolerance, run a human review sample on the classified events before feeding them into this summarization prompt. The output is a stakeholder brief, not a forensic artifact—treat it as a communication layer that sits on top of validated trace analysis, not as the analysis itself.

This prompt template is designed to be wired into a production pipeline that has already classified hallucination events from trace data. It takes structured inputs—unsupported claim counts, grounding failure categories, affected content domains, and trace timestamps—and produces a narrative report suitable for product managers, content stakeholders, and quality leads. The output prioritizes clarity over technical depth, framing hallucinations in terms of user-facing impact rather than model internals.

textCopy
You are a quality engineer writing a hallucination report for product and content stakeholders. Your audience needs to understand what types of unsupported claims occurred, how frequently, which content domains were affected, and what the business impact might be. Do not use jargon like "logits," "beam search," or "attention heads." Use plain language and concrete examples.

Generate a structured hallucination report using the following inputs:

[TRACE_SUMMARY]
[GROUNDING_FAILURE_CATEGORIES]
[AFFECTED_CONTENT_DOMAINS]
[TIMEFRAME]
[SEVERITY_THRESHOLD]

Report structure:
1. Executive Summary (2-3 sentences on overall hallucination volume and trend)
2. Hallucination Frequency (total unsupported claims, rate per 100 responses, comparison to prior period if available)
3. Failure Category Breakdown (for each category in [GROUNDING_FAILURE_CATEGORIES], include count, percentage, and one representative example from [TRACE_SUMMARY])
4. Affected Content Domains (for each domain in [AFFECTED_CONTENT_DOMAINS], include exposure count and risk level: Low/Medium/High)
5. Business Impact Assessment (user-facing consequences, trust risk, regulatory exposure if applicable)
6. Recommended Actions (2-4 concrete next steps for the product or content team)

Constraints:
- If any category exceeds [SEVERITY_THRESHOLD], flag it with **[ELEVATED]** in the report.
- If [TRACE_SUMMARY] contains fewer than 5 total events, add a note: "Low sample size; trends may not be statistically significant."
- Never fabricate examples. Use only examples present in [TRACE_SUMMARY].
- If a domain has zero events, omit it from the report rather than listing it with zero count.
- Use markdown formatting for readability.

Before sending this prompt to the model, validate that [TRACE_SUMMARY] contains actual hallucination classifications with source-evidence pairs. If your upstream hallucination detector has a known false-positive rate, include that rate as a caveat in the report preamble. For high-stakes domains such as healthcare or finance, route the generated report through a human reviewer before distribution. Adapt the severity threshold based on your organization's risk tolerance—a threshold of 5% may be appropriate for customer-facing content, while 10% may be acceptable for internal tooling. If you need to compare against a prior period, add a [PRIOR_PERIOD_DATA] placeholder and adjust the report structure to include a trend comparison section.

The Stakeholder-Friendly Hallucination Report prompt is designed to sit downstream of a hallucination detection pipeline. Before invoking this prompt, your system should already have classified individual claims as supported, unsupported, or ambiguous by comparing generated outputs against retrieved context and source documents captured in production traces. The prompt accepts a structured input payload containing pre-classified claims, grounding failure categories, affected content domains, and trace metadata. Do not feed raw, unclassified trace dumps into this prompt—it will produce unreliable summaries and may hallucinate its own classification of the underlying data.

Wire this prompt into an application as a post-processing step in a trace-analysis workflow. The recommended flow: (1) a hallucination classifier processes trace data and outputs a structured JSON array of claims with claim_text, grounding_status, failure_category, source_document_id, and content_domain fields; (2) an aggregation step groups claims by failure category and domain, computing frequency counts and trend deltas; (3) the aggregated payload is passed to this prompt as [HALLUCINATION_DATA] along with [REPORTING_PERIOD], [STAKEHOLDER_CONTEXT], and [BUSINESS_IMPACT_FRAMING] placeholders; (4) the model generates a stakeholder-ready report; (5) the output is validated against the [OUTPUT_SCHEMA] before delivery. Implement a retry loop with exponential backoff (max 3 attempts) if the output fails schema validation or if the model returns a refusal. Log every invocation—including input payload hash, model version, token counts, latency, and validation results—to your observability platform for audit and debugging.

For high-stakes reports going to executives, compliance officers, or external stakeholders, insert a mandatory human review gate after validation passes. The review interface should display the structured input data alongside the generated report so reviewers can spot-check claim-to-summary fidelity. Pay particular attention to the business_impact section of the output: if the model overstates severity or invents dollar figures, the reviewer must catch it. Consider running an LLM-as-judge eval on every generated report using a rubric that checks for hallucination classification accuracy, appropriate severity framing, and absence of fabricated metrics. If the judge score falls below your threshold (e.g., 0.85 on a 0-1 scale), flag the report for mandatory human revision rather than automatic delivery. Model choice matters here—use a model with strong instruction-following and low hallucination rates for the summarization step (Claude 3.5 Sonnet or GPT-4o are reasonable defaults), and consider using a separate, cheaper model for the judge eval to avoid self-review bias.