This prompt is for AI engineers and quality teams who operate fine-grained citation systems where the model must link specific text spans in its output to specific evidence spans in the source material. The job-to-be-done is not just checking if a citation exists, but verifying that the cited span precisely covers the claim—no more, no less. Over-citation erodes user trust by claiming support for statements the source never made; under-citation leaves supported claims looking like unsourced speculation. Use this prompt when you have a production trace that includes the generated output with its attribution spans and the full retrieved source documents, and you need a rigorous, span-level precision and recall audit.

The ideal user is a developer or quality engineer who has already instrumented their RAG pipeline to capture attribution metadata in traces. You must have access to the raw trace data: the final generated text, the array of attribution spans (each with start/end character offsets and a source document pointer), and the complete content of each cited source document. Without this granular trace data, the prompt cannot perform span-level alignment. Do not use this prompt for high-level fact-checking or binary hallucination detection—those are different jobs covered by sibling playbooks like the RAG Citation Verification Trace Prompt or the Unsupported Claim Detection Prompt. This prompt is specifically for systems where the model itself is expected to output inline citations with character-level precision.

This prompt is not a replacement for automated span-matching algorithms or deterministic string comparison. It is a diagnostic tool for cases where semantic alignment matters: when a claim is supported by a source but the model cited the wrong sentence, or when the model cited a whole paragraph for a single fact. It is also not suitable for real-time guardrails due to the latency of LLM-based review. Use it for offline trace audits, regression test suites, and periodic quality reviews. If your system does not produce attribution spans, start with a broader hallucination trace analysis prompt instead. Before running this prompt, ensure your traces are complete and your source documents are accessible—missing context will produce unreliable precision/recall scores.

This prompt template is designed to be copied directly into your AI harness for fine-grained citation review. It instructs the model to act as a forensic auditor, comparing the generated output against the retrieved source documents captured in a production trace. The core task is to evaluate every attribution span—the specific text segment linked to a citation—and determine if it accurately, completely, and exclusively represents the source material. The output is a structured report with span-level precision and recall metrics, making it suitable for automated evaluation pipelines or human-in-the-loop quality assurance workflows.

markdownCopy
You are an expert forensic auditor evaluating the precision of inline citations in AI-generated text. Your task is to compare the [GENERATED_OUTPUT] against the [RETRIEVED_SOURCES] provided in the trace to verify the accuracy of every attribution span.

An attribution span is a segment of text in the [GENERATED_OUTPUT] that is directly linked to a specific source ID via a citation marker (e.g., `[1]`). For each span, you must determine if it is:
- **Precise**: The span's claim is fully and directly supported by the cited source text. No more, no less.
- **Over-claiming**: The span makes a claim that is broader, stronger, or more specific than what the source text supports.
- **Under-claiming**: The span omits a critical part of the claim that *is* supported by the source, making the citation misleadingly narrow.
- **Unsupported**: The span's claim cannot be found in or reasonably inferred from the cited source text.

# INPUT DATA

## Generated Output with Attribution Spans
```json
[GENERATED_OUTPUT]

Retrieved Sources from Trace

jsonCopy
[RETRIEVED_SOURCES]

OUTPUT SCHEMA

Return a single JSON object conforming to this schema:

jsonCopy
{
  "spans": [
    {
      "span_id": "string (unique identifier for the span)",
      "span_text": "string (the exact text of the attribution span)",
      "cited_source_id": "string (the ID of the source it cites)",
      "status": "precise | over-claiming | under-claiming | unsupported",
      "explanation": "string (a concise, evidence-based justification for the status, quoting the source text)",
      "correction": "string | null (if not precise, provide a corrected span text that would be precise)"
    }
  ],
  "aggregate_metrics": {
    "total_spans": "number",
    "precise_count": "number",
    "over_claiming_count": "number",
    "under_claiming_count": "number",
    "unsupported_count": "number",
    "span_precision": "number (precise_count / total_spans)",
    "span_accuracy": "number ((precise_count) / (total_spans - unsupported_count))"
  },
  "trace_event_references": [
    {
      "span_id": "string",
      "trace_event_id": "string (the ID of the retrieval or generation event in the trace where this span was created)"
    }
  ]
}

CONSTRAINTS

• [CONSTRAINTS]
• If no attribution spans are present in the [GENERATED_OUTPUT], return an empty spans array and set all metrics to 0.
• Your explanation must quote the specific sentence or phrase from the source that supports or contradicts the span.
• Do not invent source content. If the source is ambiguous, mark the span as unsupported and explain the ambiguity.

To adapt this template for your system, replace the square-bracket placeholders with your actual data structures. The [GENERATED_OUTPUT] should be the final text with citation markers, and [RETRIEVED_SOURCES] should be a JSON object mapping source IDs to their full text as captured in the trace. The [CONSTRAINTS] placeholder is critical for high-risk domains; for example, in a healthcare setting, you might add: If a span makes a clinical claim, and the source only states a correlation, mark it as over-claiming. After running this prompt, always validate the output JSON against the schema before ingesting it into your monitoring dashboard. For production systems, log any parse failures and trigger a retry with a simpler, more explicit output format instruction.

This prompt is designed to be a programmatic evaluation step, not a one-off chat interaction. It expects a structured trace object containing the generated output, the list of attribution spans, and the full retrieved context for each cited source. The primary integration point is within a CI/CD pipeline for prompt releases, a scheduled batch evaluation job, or a human-in-the-loop QA tool where a reviewer can inspect flagged spans. The prompt's output—a JSON object with span-level precision and recall metrics—is intended to be consumed by a monitoring dashboard or an alerting system, not just read by a human.

To wire this into an application, construct the [TRACE_INPUT] by serializing the relevant fields from your tracing system (e.g., LangSmith, Arize, or a custom log). The input must include: the generated_text, an array of attribution_spans (each with span_id, text, and cited_source_id), and a map of retrieved_sources keyed by source_id with their full text. Before calling the model, validate the input shape against a strict schema (e.g., using Pydantic or JSON Schema) to prevent malformed requests. After receiving the model's output, validate the JSON structure and check for the span_evaluations array. Implement a retry loop with a maximum of 2 attempts if the output fails to parse or is missing required fields. Log every raw request and response pair, along with the parsed metrics, to your observability platform for later analysis.

For high-stakes applications, do not treat the model's precision/recall scores as ground truth without verification. Implement a calibration step where a random sample of span evaluations is reviewed by a human, and the model's agreement rate with the human is tracked over time. If the agreement rate drops below a threshold (e.g., 90%), pause automated gating and escalate for prompt or model review. Use a capable model with strong instruction-following and JSON mode (such as GPT-4o or Claude 3.5 Sonnet) for this task; smaller or older models often struggle with the precise text-boundary matching required for span-level attribution. Finally, feed the per-span precision and recall metrics into your evaluation dashboard, setting alerts for any regression below a predefined threshold (e.g., precision < 0.95) to catch citation quality degradation before it reaches users.