Prompt Version vs Hallucination Rate Trace Prompt

This prompt is for prompt engineers and AI reliability engineers who need to answer a specific question before a release: 'Did the new prompt version cause the model to hallucinate more?' It is not a real-time monitor, a single-trace debugger, or a general-purpose evaluation rubric. It is a batch comparison tool that takes production traces already labeled for hallucination, groups them by prompt version and query cluster, and produces a structured regression report. The ideal user already has a trace dataset with hallucination labels, version identifiers, and query embeddings or cluster assignments. If you are missing any of these three inputs, this prompt will not produce reliable results—you need to instrument your production pipeline first.

Use this prompt during three specific workflow stages: canary releases, where a new prompt version serves a small percentage of traffic and you need to compare hallucination rates against the stable version before expanding; pre-promotion gates, where you run a final statistical check on a holdout trace set before switching 100% of traffic; and incident investigation, where a suspected hallucination spike after a prompt change requires a controlled comparison that separates the prompt effect from a shift in user queries. The prompt controls for query distribution shifts by stratifying comparisons within query clusters, so a sudden influx of harder questions won't masquerade as a prompt regression. It also categorizes severity so you can distinguish a critical safety regression from a minor precision dip.

Do not use this prompt when you lack per-trace hallucination labels, when you have fewer than a few hundred traces per version, or when your query clusters are unstable. Statistical significance requires sufficient sample size, and the prompt will flag low-confidence comparisons rather than fabricate certainty. This prompt also does not replace your real-time monitoring dashboard—it is a point-in-time analysis, not a continuous guardrail. After running this analysis, the next step is typically to drill into flagged clusters with a root-cause trace analysis prompt to identify whether the regression stems from instruction leakage, context truncation, or retrieval failure. If the regression is severe, block the promotion and loop in your RAG and search engineers.

This prompt is designed to be pasted directly into your trace analysis pipeline. It instructs the model to act as a statistical reviewer, comparing two prompt versions by analyzing a batch of production traces. The template uses square-bracket placeholders for all dynamic inputs, such as trace data, version identifiers, and statistical thresholds. Before using this prompt, ensure your trace data has been pre-processed to include the final output, the retrieved context, and a unique trace ID for each session. The prompt's primary job is to produce a structured regression report, not to execute the statistical test itself; you should validate the model's arithmetic and reasoning against a known statistical library in your application harness.

textCopy
You are an AI reliability engineer reviewing production traces to compare hallucination rates between two prompt versions. Your analysis must be grounded only in the provided trace data. Do not invent or assume any information not present in the traces.

You will receive a set of traces from two prompt versions: [PROMPT_VERSION_A] and [PROMPT_VERSION_B]. For each trace, you are given the [TRACE_ID], the [USER_QUERY], the [RETRIEVED_CONTEXT], and the [GENERATED_OUTPUT].

Your task is to:
1. For each trace, determine if the [GENERATED_OUTPUT] contains any factual claim that is not supported by the [RETRIEVED_CONTEXT]. A claim is unsupported if it introduces a new entity, number, date, or relationship not present in the context.
2. Classify each trace as 'hallucination_detected' or 'no_hallucination_detected'.
3. Calculate the hallucination rate for [PROMPT_VERSION_A] and [PROMPT_VERSION_B] as the percentage of traces with a detected hallucination.
4. Compare the two rates. Flag the difference as 'statistically_significant' if the absolute percentage-point difference exceeds [SIGNIFICANCE_THRESHOLD_PCT].
5. If a significant change is detected, identify the top three categories of hallucination that changed the most (e.g., 'fabricated_numbers', 'false_relationships', 'overgeneralization').

Your output must be a single JSON object conforming to this schema:
{
  "analysis_id": "string",
  "prompt_version_a": "string",
  "prompt_version_b": "string",
  "total_traces_analyzed": "integer",
  "version_a_hallucination_rate_pct": "float",
  "version_b_hallucination_rate_pct": "float",
  "absolute_rate_change_pct": "float",
  "is_change_statistically_significant": "boolean",
  "significance_threshold_used_pct": "float",
  "top_changed_hallucination_categories": [
    {
      "category": "string",
      "version_a_count": "integer",
      "version_b_count": "integer",
      "change_description": "string"
    }
  ],
  "per_trace_results": [
    {
      "trace_id": "string",
      "prompt_version": "string",
      "hallucination_status": "hallucination_detected | no_hallucination_detected",
      "unsupported_claim_summary": "string or null"
    }
  ]
}

Here is the trace data to analyze:
[TRACE_DATA_JSON]

To adapt this template, replace the bracketed placeholders with your specific data. [PROMPT_VERSION_A] and [PROMPT_VERSION_B] should be the exact identifiers from your prompt registry. [SIGNIFICANCE_THRESHOLD_PCT] is a business decision based on your risk tolerance; a common starting point is 5.0. The [TRACE_DATA_JSON] placeholder must be replaced with a valid JSON array of trace objects, each containing the required fields. If your traces are large, consider pre-processing them to include only the most relevant retrieved passages to avoid exceeding the model's context window. Always run this prompt with a low temperature setting (e.g., 0.0 or 0.1) to maximize deterministic, repeatable analysis.

This prompt is designed to be run as a scheduled batch job, not a real-time request. It consumes a structured dataset of production traces sampled across two prompt versions, applies statistical tests, and produces a grounding regression report. The implementation harness must handle trace extraction, data normalization, prompt execution, output validation, and alert routing. Because the output may trigger a prompt rollback decision, the harness must be auditable and the prompt's reasoning must be traceable back to specific trace events.

The harness should query your observability store (e.g., LangSmith, Arize, Braintrust, or a custom trace warehouse) for traces matching the two prompt version identifiers in [PROMPT_VERSION_A] and [PROMPT_VERSION_B]. For each trace, extract: the final generated output, the retrieved context chunks with their trace span IDs, any user feedback signals, and the hallucination label if already scored by an evaluator. Normalize this into a JSON array matching the [TRACE_BATCH] schema before inserting it into the prompt. After execution, validate the output against [OUTPUT_SCHEMA] using a strict JSON Schema validator. If validation fails, retry once with a repair prompt that includes the schema violation errors. Log every report to a versioned audit table with the run timestamp, prompt version IDs, sample sizes, and the full report payload. If the report flags a statistically significant regression in grounding_regression_detected, route the report to the on-call channel and the prompt registry's release gate.

Do not run this prompt on traces that lack retrieval context—it will produce meaningless results. Ensure your trace extraction query filters for RAG sessions only. If your observability store does not natively capture hallucination labels, run a separate faithfulness evaluator on the sampled traces first and include those scores in the batch. The statistical tests in this prompt assume independent samples; if your trace sampling method introduces session-level correlation, note this in the report's caveats or pre-aggregate to one trace per session. Finally, treat this prompt's output as a diagnostic signal, not an automated rollback trigger—always require human review of the report before reverting a prompt version in production.