Hallucination Rate Monitoring Query Prompt Template

This prompt is for quality engineers and AI operations teams who need to move from ad-hoc trace review to continuous, query-driven hallucination monitoring. The job-to-be-done is generating a precise, executable monitoring query that can detect, score, and trend unsupported claims across production traces. The ideal user has access to a trace observability platform containing groundedness evaluation results and needs to define alertable conditions for hallucination rates without manually inspecting every session.

Use this prompt when you have structured groundedness eval results already attached to your production traces and you need to define a repeatable query for a dashboard or alerting system. The prompt requires [GROUNDEDNESS_EVAL_SCHEMA], [TRACE_SOURCE], and [MONITORING_WINDOW] as inputs. It is not a replacement for the groundedness evaluation itself—it assumes that eval has already run and produced scores per span or per response. Do not use this prompt to perform the initial fact-checking or to adjudicate individual claims; it is designed for aggregate monitoring, not single-case diagnosis.

Before using this prompt, ensure your trace data includes fields for claim-level groundedness scores, source document identifiers, and response IDs. If your groundedness eval only produces a binary pass/fail, you will need to enrich it with confidence scores or detailed labels before this prompt can produce meaningful distributions and trends. The output is a monitoring query, not a final report—you should validate the query against a known trace dataset, tune the alert thresholds, and integrate the query into your observability platform with a runbook for what to do when the hallucination rate breaches the defined threshold.

This prompt template is designed to be wired into a monitoring pipeline that consumes production traces and groundedness evaluation results. It instructs the model to generate structured queries, score distributions, and trend analyses that can be fed into dashboards or alerting systems. The template uses square-bracket placeholders for all dynamic inputs, making it safe to template in code before sending to the model.

textCopy
You are a production quality engineer analyzing AI system traces for hallucination rates. Your task is to generate monitoring queries and analyses based on the provided groundedness evaluation results.

## INPUT DATA
[TRACE_DATA]

## EVALUATION CONTEXT
- Groundedness Eval Results: [GROUNDEDNESS_EVAL_RESULTS]
- Monitoring Time Window: [TIME_WINDOW]
- Alert Threshold Configuration: [ALERT_THRESHOLDS]
- Baseline Hallucination Rate: [BASELINE_RATE]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "hallucination_score_distribution": {
    "bins": [{"range": "string", "count": integer, "percentage": float}],
    "summary_statistics": {
      "mean": float,
      "median": float,
      "p95": float,
      "p99": float
    }
  },
  "grounding_gap_identification": [
    {
      "trace_id": "string",
      "claim": "string",
      "gap_type": "unsupported_claim|citation_mismatch|evidence_missing|fabricated_source",
      "severity": "critical|high|medium|low",
      "affected_span": "string"
    }
  ],
  "trend_detection": {
    "current_rate": float,
    "baseline_comparison": {
      "delta": float,
      "direction": "increasing|decreasing|stable",
      "statistical_significance": "significant|not_significant|insufficient_data"
    },
    "time_series": [{"timestamp": "string", "rate": float}],
    "anomaly_flags": ["string"]
  },
  "alert_evaluation": {
    "threshold_breaches": [
      {
        "threshold_name": "string",
        "current_value": float,
        "threshold_value": float,
        "breach_severity": "critical|warning|info"
      }
    ],
    "recommended_actions": ["string"]
  },
  "query_suggestions": [
    {
      "query_name": "string",
      "query_description": "string",
      "query_logic": "string",
      "intended_dashboard": "string"
    }
  ]
}

## CONSTRAINTS
- Only flag claims as hallucinations when the groundedness eval score is below [GROUNDEDNESS_THRESHOLD].
- Do not fabricate trace IDs or eval scores not present in the input data.
- Mark any analysis requiring unavailable data as "insufficient_data".
- For trend detection, require a minimum of [MIN_SAMPLE_SIZE] data points before declaring statistical significance.
- If the input data is empty or malformed, return a structured error object with a clear message.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Generate the monitoring analysis now.

To adapt this template, replace each square-bracket placeholder with values from your observability pipeline. The [TRACE_DATA] placeholder expects structured trace spans with groundedness eval scores attached. The [GROUNDEDNESS_EVAL_RESULTS] should contain per-claim scores from your evaluation system. For high-risk production environments where hallucination alerts trigger automatic rollbacks or paging, set [RISK_LEVEL] to "high" and ensure the output passes through a human review step before automated actions execute. The [FEW_SHOT_EXAMPLES] placeholder should contain 2-3 example input-output pairs that demonstrate correct gap identification and threshold evaluation for your specific domain. Always validate the model's output against the defined JSON schema before ingestion into monitoring dashboards, and log any schema violations for prompt refinement.

This prompt is designed to be executed as part of a scheduled batch job, not an interactive chat. The primary integration point is a trace analytics backend (e.g., LangSmith, Arize, Datadog LLM Observability, or a custom OpenTelemetry pipeline) that stores groundedness eval scores as span attributes. The application harness should run the query on a fixed cadence—hourly for high-traffic systems, daily for lower-volume products—and feed the results into a structured alerting rule. The prompt itself is stateless; all context comes from the [TRACE_DATA] input, which must be pre-aggregated from the observability store before invocation.

The harness requires a preprocessing step that queries the trace store for spans with a groundedness_score attribute within the [TIME_WINDOW]. This step must filter out incomplete traces and normalize scores to a 0–1 range. The preprocessed payload should include per-span metadata: trace_id, timestamp, groundedness_score, source_documents (if available), and the generated output_text. This structured input is what gets injected into the [TRACE_DATA] placeholder. After the model returns the analysis, a post-processing validator must check that the output conforms to the [OUTPUT_SCHEMA]—specifically that hallucination_rate is a float between 0 and 1, trend_direction is one of the allowed enum values, and alert_triggered is a boolean. If validation fails, the harness should retry once with the same input and a stronger format constraint appended to the prompt. If the retry also fails, log the raw output and escalate to the on-call channel without blocking the monitoring pipeline.

Model choice matters here. Use a model with strong JSON mode support and a large enough context window to hold the full [TRACE_DATA] payload without truncation. For high-cardinality trace volumes, consider batching traces by session or sampling before sending to the model. Wire the alert_triggered boolean into your alerting platform (PagerDuty, Opsgenie, or a Slack webhook) with the alert_summary as the notification body. Store the full structured output in a time-series database alongside the raw trace batch ID for auditability. Avoid using this prompt for real-time per-request hallucination checks—it is designed for aggregate trend detection, not inline gating. For inline use, a lightweight groundedness eval scorer at request time is more appropriate.