This prompt is designed for AI observability platforms and site reliability engineers who need continuous, automated monitoring of factual accuracy in production model outputs. Use it when you have a stream of logged model responses paired with their source context and need to generate periodic accuracy trend reports. The prompt performs claim extraction, evidence matching, hallucination classification, and metric aggregation across a batch of samples. It is not a single-response fact checker. It is a batch monitoring instrument that produces structured reports suitable for dashboard ingestion, alerting thresholds, and accuracy degradation detection. Deploy this when you need to move from spot-checking outputs to systematic production accuracy surveillance.
Prompt
Production Fact Accuracy Monitoring Prompt
When to Use This Prompt
Identifies the operational context, required inputs, and boundary conditions for deploying a batch production fact accuracy monitor.
The ideal input is a structured batch of records, each containing a [MODEL_OUTPUT] and its corresponding [SOURCE_CONTEXT]. The prompt expects these pairs to be pre-collected from your logging pipeline. It is not designed for real-time, single-turn user interactions. The output is a machine-readable JSON report containing per-sample hallucination flags, an aggregated hallucination rate, a severity distribution, and a list of failure patterns. This output is intended to be consumed by a monitoring dashboard or an alerting system, not displayed directly to end users. You should configure your harness to run this prompt on a scheduled basis (e.g., hourly or daily) against a statistically significant sample of recent traffic.
Do not use this prompt as a real-time guardrail that blocks responses. Its strength is in trend analysis and degradation detection, not low-latency intervention. It is also inappropriate when source context is unavailable or when the model's task is creative generation rather than factual synthesis. Before deploying, ensure you have a reliable mechanism for pairing outputs with their source context in your logging pipeline. The next step is to review the prompt template and adapt the [OUTPUT_SCHEMA] to match your specific monitoring backend.
Use Case Fit
Where this prompt works and where it does not. Understand the operational boundaries before wiring it into your production harness.
Good Fit: Automated Observability Pipelines
Use when: You have a production observability pipeline that logs model inputs, retrieved context, and final outputs. This prompt excels at sampling those logs and performing automated, claim-by-by-claim audits to generate accuracy trend reports. Guardrail: Ensure the prompt has access to the original, unmodified context used during generation, not a re-retrieved set that may differ.
Bad Fit: Real-Time Guardrails
Avoid when: You need to block a hallucinated response before it reaches the user. This prompt is designed for post-hoc analysis and trend reporting, not for low-latency, synchronous validation in the critical path. Guardrail: Use a lightweight, high-speed fact-checking prompt for real-time guardrails and reserve this prompt for asynchronous monitoring and alerting.
Required Inputs: Logged Context is Mandatory
Risk: Running this prompt without the exact source context that was available during generation will produce unreliable and misleading accuracy scores. Guardrail: Your logging pipeline must capture and store the full prompt, retrieved chunks, and final output as an atomic unit. The monitoring job must fetch this tuple before invoking the prompt.
Operational Risk: Silent Hallucination Drift
Risk: A slow degradation in retrieval quality or a model update can cause a gradual increase in hallucination rates that goes unnoticed without trend analysis. Guardrail: Configure this prompt to run on a scheduled basis (e.g., daily) and set statistical alert thresholds on the hallucination rate metric to detect drift before it becomes a user-facing incident.
Operational Risk: Costly Full-Corpus Audits
Risk: Running this detailed, claim-by-claim audit on 100% of production traffic is prohibitively expensive and slow. Guardrail: Implement a statistically significant sampling strategy (e.g., random sampling, stratified sampling by topic) to keep costs predictable while maintaining a reliable accuracy signal. Never audit everything by default.
Bad Fit: Unstructured or Missing Source Grounding
Avoid when: Your application generates creative text, performs summarization without retained context, or operates in a setting where the 'ground truth' is subjective. This prompt requires a verifiable source of truth to function. Guardrail: Only deploy this prompt for RAG, document Q&A, and other workflows where a definitive context document exists. For creative tasks, use a different evaluation rubric.
Copy-Ready Prompt Template
A reusable prompt template for sampling production outputs and performing automated fact-checking against logged context to generate accuracy trend reports.
This template is the core instruction payload for a production fact accuracy monitoring harness. It is designed to be invoked on a scheduled basis or triggered by a sampling pipeline. The prompt instructs the model to act as an automated auditor: it receives a batch of production interactions—each containing the original user query, the retrieved or provided context, and the final model response—and performs a systematic, claim-by-claim verification. The output is a structured accuracy report suitable for ingestion into observability dashboards, alerting systems, and regression test suites. Before using this template, ensure your sampling logic captures the full context window that was available at generation time; auditing against incomplete context produces false hallucination flags.
textYou are a production fact accuracy auditor for an AI system. Your task is to analyze a batch of production interactions and produce a structured accuracy report. ## INPUT You will receive a JSON array of interaction records. Each record contains: - `interaction_id`: unique identifier - `timestamp`: ISO 8601 timestamp of the original interaction - `user_query`: the user's original input - `retrieved_context`: the source text provided to the model at generation time - `model_response`: the final response delivered to the user [INPUT_BATCH] ## TASK For each interaction, perform the following steps: 1. **Claim Extraction**: Extract every discrete factual assertion from `model_response`. Ignore opinions, stylistic language, and polite filler. A factual assertion is a statement that can be verified against `retrieved_context`. 2. **Evidence Matching**: For each extracted claim, search `retrieved_context` for supporting evidence. A claim is **supported** if the context directly states it. A claim is **unsupported** if the context does not contain the information. A claim is **contradicted** if the context states the opposite. 3. **Classification**: Assign each claim one of the following labels: - `SUPPORTED`: directly backed by retrieved_context - `UNSUPPORTED`: no evidence in retrieved_context (potential hallucination) - `CONTRADICTED`: context states the opposite (definite hallucination) - `OUT_OF_SCOPE`: claim is about the world, not the context, and cannot be verified 4. **Severity Assessment**: For each UNSUPPORTED or CONTRADICTED claim, assign a severity: - `CRITICAL`: claim would mislead the user on a safety, compliance, or high-stakes matter - `HIGH`: claim introduces a material factual error - `MEDIUM`: claim is minor or tangential - `LOW`: claim is trivial or would not affect user decisions ## OUTPUT SCHEMA Return a JSON object conforming to this schema: [OUTPUT_SCHEMA] ## CONSTRAINTS - Do not fabricate evidence. If retrieved_context does not support a claim, mark it UNSUPPORTED. - Do not use external knowledge. Verify only against the provided retrieved_context. - If retrieved_context is empty or missing, mark all factual claims as UNSUPPORTED. - If a claim is partially supported, mark it UNSUPPORTED and note what is missing. - Include the exact text spans from retrieved_context that support SUPPORTED claims. - Include the exact text spans from model_response for each claim. - For CONTRADICTED claims, include the conflicting text from retrieved_context. ## EXAMPLES [EXAMPLES] ## RISK LEVEL [RISK_LEVEL]
To adapt this template for your production environment, replace each square-bracket placeholder with concrete values before execution. [INPUT_BATCH] should be populated with a JSON array of interaction records from your sampling pipeline—ensure each record includes the full retrieved_context that was available at generation time, not a truncated or post-hoc version. [OUTPUT_SCHEMA] must be replaced with your exact expected JSON schema, including field types, required fields, and enum values for claim labels and severity levels. [EXAMPLES] should contain at least two few-shot examples: one showing a clean interaction with all claims supported, and one showing a mix of supported, unsupported, and contradicted claims with correct severity assignments. [RISK_LEVEL] should be set to high for regulated domains (healthcare, legal, finance) where hallucination carries compliance risk, or medium for general product use. If your monitoring pipeline runs on a schedule, inject the time window and sampling parameters as additional context before the input batch to help the model interpret trends. Always validate the output against your schema before ingestion; malformed JSON or missing required fields should trigger a retry with a repair prompt, not silent acceptance.
Prompt Variables
Required inputs for the Production Fact Accuracy Monitoring Prompt. Each variable must be populated before the prompt is assembled and sent. Missing or malformed variables will cause the monitoring pipeline to fail silently or produce unreliable accuracy metrics.
| Placeholder | Purpose | Example | Validation Notes |
|---|---|---|---|
[SAMPLED_OUTPUTS] | Batch of production model responses to audit for factual accuracy | A JSON array of 50 response objects, each containing response_id, full_text, and generation_timestamp | Must be a valid JSON array with 10-200 objects. Each object requires response_id (string), full_text (non-empty string), and generation_timestamp (ISO 8601). Reject if array is empty or exceeds 200 items. |
[SOURCE_CONTEXT] | The original context or evidence provided to the model when generating each sampled output | A JSON object mapping response_id to the array of retrieved passages, documents, or tool outputs available at generation time | Must be a valid JSON object. Each key must match a response_id in [SAMPLED_OUTPUTS]. Each value must be a non-empty array of context strings. Reject if any response_id is missing context or context array is empty. |
[FACT_CHECKING_RUBRIC] | Scoring criteria defining what constitutes a supported fact, inference, or hallucination | A structured rubric with definitions for supported_claim, unsupported_claim, fabrication, and abstention, plus severity weights | Must be a valid JSON object with required keys: supported_claim, unsupported_claim, fabrication, abstention. Each key must have a definition (string) and severity_weight (number 0.0-1.0). Reject if weights don't sum to 1.0 or definitions are empty. |
[ALERT_THRESHOLDS] | Numerical thresholds that trigger accuracy degradation alerts | hallucination_rate: 0.05, fabrication_rate: 0.01, unsupported_claim_rate: 0.10, severity_weighted_score_min: 0.92 | Must be a valid JSON object with required numeric fields: hallucination_rate, fabrication_rate, unsupported_claim_rate, severity_weighted_score_min. All values must be between 0.0 and 1.0. Reject if any threshold is negative or exceeds 1.0. |
[OBSERVABILITY_WINDOW] | Time range for the accuracy monitoring report | start_time: 2025-01-15T00:00:00Z, end_time: 2025-01-22T00:00:00Z, granularity: daily | Must be a valid JSON object with start_time and end_time as ISO 8601 strings, and granularity as one of hourly, daily, weekly. End time must be after start time. Window must not exceed 30 days. Reject if format is invalid or window exceeds limit. |
[MODEL_IDENTIFIER] | Identifier for the model version being monitored, used for trend comparison | gpt-4o-2024-11-20 or claude-sonnet-4-20250514 | Must be a non-empty string matching the organization's model registry format. Validate against allowed model identifiers list. Reject if identifier is unknown or deprecated. |
[PREVIOUS_REPORT_ID] | Optional reference to a prior accuracy report for trend comparison and regression detection | report_2025-01-15_weekly or null | If provided, must be a non-empty string matching the report ID format. If null, trend comparison is skipped. Validate that referenced report exists in the observability store before processing. Reject if report ID is malformed or references a future window. |
[OUTPUT_FORMAT] | Desired structure for the accuracy trend report | json_schema_version: 2.1, include_claim_details: true, include_pattern_clusters: true, max_failure_examples: 20 | Must be a valid JSON object with required fields: json_schema_version (string), include_claim_details (boolean), include_pattern_clusters (boolean), max_failure_examples (integer 0-100). Reject if max_failure_examples exceeds 100 or required fields are missing. |
Implementation Harness Notes
How to wire the Production Fact Accuracy Monitoring Prompt into an observability pipeline with validation, retries, and alerting.
This prompt is designed to run as a scheduled or event-driven job inside your AI observability stack, not as a real-time user-facing call. It expects batched samples of production outputs paired with their logged source context. The harness should pull these pairs from your logging backend (e.g., a database of stored completions and their retrieval-augmented generation context), format them into the [OUTPUT_SAMPLE_BATCH] placeholder, and invoke the model. Because this is a monitoring workflow, the prompt's output is a structured accuracy report, not a corrected answer. Your harness must parse this report and route it to your metrics store and alerting system.
Implement the call with a strict JSON mode or structured output API to guarantee the report schema. Wrap the invocation in a retry loop with a small budget (2-3 attempts) for transient model failures or malformed JSON. After parsing, run a schema validator against the expected fields: hallucination_rate, severity_distribution, failure_patterns, and alert_threshold_breaches. If validation fails after retries, log the raw output and raise a non-critical monitoring alert—do not silently drop the sample. For high-risk domains, route reports with severity_distribution.critical > 0 to a human review queue before the alert fires. Store the structured report in your time-series metrics database and attach the raw prompt and response to the trace for later audit.
Set your alert thresholds in the application layer, not in the prompt. The prompt reports the metrics; your harness compares hallucination_rate against a configured threshold (e.g., >5% triggers a warning, >10% triggers a critical alert). Use the failure_patterns array to populate a dashboard that clusters recurring hallucination types—entity confusion, temporal errors, quantity fabrication—so the team can target specific prompt or retrieval fixes. Avoid running this prompt on every single production output; sample at a rate that balances coverage with cost (e.g., 5-10% of traffic, or all outputs flagged by a cheaper upstream confidence scorer). If the prompt itself fails repeatedly, escalate to the on-call engineer and investigate whether the model endpoint, context format, or sampling logic has drifted.
Expected Output Contract
Defines the schema, types, and validation rules for the structured monitoring report produced by the Production Fact Accuracy Monitoring Prompt. Use this contract to build downstream parsers, dashboards, and alerting logic.
| Field or Element | Type or Format | Required | Validation Rule |
|---|---|---|---|
report_id | string (UUID v4) | Must match UUID v4 regex. Auto-generated if missing. | |
report_timestamp | string (ISO 8601) | Must parse as valid ISO 8601 datetime. Reject if in the future by more than 5 minutes. | |
sample_window | object | Must contain | |
total_samples_evaluated | integer | Must be a positive integer. Must equal the sum of | |
accuracy_summary | object | Must contain integer fields | |
hallucination_rate | number (float) | Must be between 0.0 and 1.0 inclusive. Calculated as | |
failure_patterns | array of objects | Each object must have | |
alert_thresholds_breached | array of strings | Each string must match a predefined alert name from [ALERT_CATALOG]. Empty array allowed. Null not allowed. |
Common Failure Modes
Fact accuracy monitoring in production breaks in predictable ways. These are the most common failure modes and the guardrails that catch them before they reach users.
Context Drift Between Logging and Verification
What to watch: The context used during generation differs from the context logged for later fact-checking due to async updates, cache invalidation, or retrieval non-determinism. The monitor flags claims as hallucinated when they were actually grounded at generation time. Guardrail: Log a content hash of the retrieved context alongside each generation. The fact-checking step compares hashes before running verification and skips audit if context has changed, logging a drift event instead.
Claim Extraction Misses Compound Statements
What to watch: The claim extraction step splits on sentence boundaries but misses compound claims within a single sentence, such as 'The patient has diabetes and was prescribed metformin in 2022.' If only the first clause is verified, the second clause escapes audit. Guardrail: Use a dedicated claim decomposition prompt that explicitly breaks sentences into atomic, independently verifiable assertions. Validate that each extracted claim contains exactly one fact before proceeding to verification.
Verification Prompt Accepts Near-Match as Confirmation
What to watch: The verification step returns 'supported' when evidence is semantically similar but factually different, such as confirming 'revenue grew 12%' when the source says 'revenue grew 12.4%.' Small numeric or entity mismatches pass through undetected. Guardrail: Include explicit numeric and entity precision rules in the verification prompt. Require exact match for numbers, dates, and proper nouns. Flag any deviation as 'partial match' rather than 'supported,' with a separate threshold for acceptable precision.
Hallucination Rate Metrics Mask Severity Differences
What to watch: A 2% hallucination rate looks acceptable in aggregate, but all hallucinations cluster in high-risk claim types such as drug dosages, legal obligations, or financial figures. Aggregate metrics hide dangerous concentration. Guardrail: Segment hallucination rate by claim category, entity type, and risk tier. Set separate alert thresholds per segment. A 1% hallucination rate on high-risk claims triggers immediate escalation even when the aggregate rate is below threshold.
Monitor Blindness to Systematic Source Errors
What to watch: The fact-checking monitor treats the logged context as ground truth, but the context itself contains errors from upstream retrieval, outdated documents, or ingestion bugs. The monitor certifies outputs as accurate when both the generation and the source are wrong in the same way. Guardrail: Periodically sample source documents and run independent quality checks outside the generation pipeline. Cross-reference claims against authoritative external sources for a random subset of audits. Flag source quality degradation as a separate alert category.
Alert Thresholds Ignore Volume Spikes
What to watch: A static hallucination rate threshold of 5% fails to catch a sudden spike from 1% to 4% that signals a retrieval pipeline regression or model update issue. The degradation persists for hours before crossing the threshold. Guardrail: Implement rate-of-change alerts alongside absolute thresholds. Trigger investigation when the hallucination rate doubles within a rolling window, even if the absolute rate remains below the static threshold. Pair with deployment-event correlation to accelerate root cause identification.
Evaluation Rubric
Criteria for testing the Production Fact Accuracy Monitoring Prompt before deployment. Each row defines a pass standard, a failure signal, and a concrete test method to validate monitoring output quality.
| Criterion | Pass Standard | Failure Signal | Test Method |
|---|---|---|---|
Hallucination Rate Calculation | Reported rate matches manual audit within ±2 percentage points on a 100-sample batch | Reported rate deviates from manual audit by more than 5 percentage points | Run prompt on a golden set of 100 pre-labeled outputs with known hallucination counts; compare reported rate to ground-truth rate |
Claim Extraction Completeness | At least 95% of discrete factual claims in the sampled output are extracted for verification | More than 10% of claims are missed, or extraction includes non-factual statements as claims | Use a curated set of 20 outputs with pre-annotated claim spans; compute recall against annotations |
Source-Context Mapping Accuracy | Each extracted claim is mapped to the correct source context chunk with at least 90% precision | Claims are mapped to wrong context chunks, or mapping is missing for more than 15% of claims | Provide outputs with single-source context; verify that every claim maps to the only available source |
Unsupported Claim Flagging | All claims with no supporting evidence in context are flagged as unsupported; no supported claims are incorrectly flagged | Supported claims are flagged as unsupported, or fabricated claims pass without flagging | Inject 5 outputs with known unsupported claims into a batch; verify all 5 are flagged and no supported claims are flagged |
Trend Report Structure Validity | Output is valid JSON matching [OUTPUT_SCHEMA] with all required fields present and correctly typed | Output is malformed JSON, missing required fields, or contains fields with wrong data types | Parse output with schema validator; confirm all required fields exist and match declared types |
Alert Threshold Triggering | Alert is raised when hallucination rate exceeds configured threshold in [ALERT_CONFIG]; no alert when rate is below threshold | Alert fires when rate is below threshold, or no alert fires when rate exceeds threshold | Run prompt on two batches: one with hallucination rate above threshold, one below; verify alert presence/absence matches threshold |
Failure Pattern Clustering | Clusters contain at least 3 outputs each and cluster labels describe a specific hallucination pattern | Clusters are empty, contain only 1 output, or labels are generic and uninformative | Feed a batch with 3 known pattern groups of 5 outputs each; verify clusters match known groups and labels are descriptive |
Retry Budget Exhaustion Handling | When retry budget is exhausted, output includes escalation record with failure summary and recommended human intervention path | Retry exhaustion produces empty output, infinite loop, or missing escalation record | Simulate context where all retries fail; verify output contains structured escalation record with failure summary and intervention recommendation |
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
Adapt This Prompt
How to adapt
Start with the base prompt and a small sample of production outputs. Use a frontier model with default temperature. Focus on getting the claim extraction and evidence-matching logic right before adding monitoring infrastructure.
Simplify the output schema to essential fields only:
code[CLAIM] | [EVIDENCE_MATCH] | [VERDICT]
Run against 20-50 historical outputs to validate the detection pattern catches obvious hallucinations.
Watch for
- Over-flagging stylistic choices as hallucinations
- Missing temporal claims (dates, sequences)
- False positives on reasonable inferences from context
- No baseline accuracy rate established before tuning
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us