Citation-to-Source Alignment Audit Prompt

This prompt is designed for compliance officers, quality engineers, and AI platform teams who need to verify that every inline citation in a generated answer is traceable to a real source document captured in the production trace. The primary job-to-be-done is producing a structured alignment scorecard that maps each citation to a specific trace event ID, confirms whether the cited content actually supports the claim, and flags misalignments for remediation. Use this prompt when you are operating a Retrieval-Augmented Generation (RAG) system in a regulated domain, preparing for an external audit, or establishing a continuous grounding quality metric. The ideal user has access to a full observability trace that includes the final generated output, the list of retrieved source documents, and their associated trace span or event IDs.

Do not use this prompt for real-time guardrailing or blocking of model outputs; it is a post-hoc diagnostic and audit tool, not a low-latency safety classifier. It is also inappropriate for evaluating the semantic quality or helpfulness of an answer—its scope is strictly limited to citation-to-source alignment. The prompt requires structured input: a generated answer containing inline citations, a set of retrieved source documents with unique identifiers, and the trace event IDs that link retrieval events to the final generation. Without these inputs, the alignment audit cannot produce a reliable scorecard. The output is a machine-readable JSON report, making it suitable for integration into automated evaluation pipelines, compliance dashboards, or periodic audit workflows.

Before running this prompt, ensure your trace data is complete and that citation markers in the generated answer are parseable. If your system uses numeric citation brackets like [1], [2], or author-date references, you must include a mapping to trace event IDs in the input context. The prompt will flag citations that point to missing documents, documents that were retrieved but not used, and claims that cite a source but are not actually supported by its content. After receiving the alignment scorecard, route any critical misalignments to a human reviewer for final determination, especially in high-risk domains like healthcare, legal, or finance. Do not treat the model's alignment judgment as ground truth without human verification when compliance consequences are material.

This prompt template is designed to be dropped into an evaluation harness or a manual review workflow. It instructs the model to compare every inline citation in a generated answer against the source documents captured in the production trace, producing a structured scorecard. The template uses square-bracket placeholders for all variable inputs, making it straightforward to wire into a CI/CD pipeline, a monitoring job, or a human-in-the-loop review queue.

textCopy
You are an auditor reviewing a generated answer and its full production trace. Your task is to verify that every inline citation in the answer maps to a real source document present in the trace and that the cited content actually supports the claim it is attached to.

## INPUT

**Generated Answer:**
[GENERATED_ANSWER]

**Trace Events (JSON array of retrieval and generation steps):**
[TRACE_EVENTS]

**Source Documents (full text keyed by document ID):**
[SOURCE_DOCUMENTS]

## OUTPUT SCHEMA

Return a valid JSON object with the following structure:
{
  "alignment_scorecard": {
    "total_citations": <integer>,
    "aligned_citations": <integer>,
    "misaligned_citations": <integer>,
    "unverifiable_citations": <integer>,
    "overall_alignment_score": <float between 0.0 and 1.0>
  },
  "misalignment_instances": [
    {
      "citation_text": "<the exact citation marker or text from the answer>",
      "claimed_source_id": "<the document ID the citation points to>",
      "trace_event_id": "<the trace event ID where this citation was generated>",
      "failure_type": "<MISSING_SOURCE | CONTENT_MISMATCH | FABRICATED_CITATION | STALE_SOURCE | AMBIGUOUS_ATTRIBUTION>",
      "failure_description": "<specific explanation of what went wrong>",
      "cited_content": "<the relevant excerpt from the source document, or null if source missing>",
      "claim_made": "<the claim the citation was supposed to support>",
      "severity": "<CRITICAL | MAJOR | MINOR>"
    }
  ],
  "unverifiable_notes": "<explanation for any citations that could not be verified due to trace gaps or missing documents>"
}

## CONSTRAINTS

1. Extract every inline citation from the generated answer. Citations may appear as bracketed numbers [1], [2], parenthetical references (Source A), or inline links.
2. For each citation, locate the corresponding source document in [SOURCE_DOCUMENTS] using the document ID referenced in the trace event.
3. Compare the claim the citation is attached to against the full text of the source document. The citation is aligned only if the source document explicitly supports the specific claim.
4. Classify each misalignment using the failure types defined in the output schema.
5. If a citation references a document ID not present in [SOURCE_DOCUMENTS], classify it as FABRICATED_CITATION.
6. If the source document exists but does not contain information supporting the claim, classify it as CONTENT_MISMATCH.
7. If the source document exists but is timestamped before a known factual update relevant to the claim, classify it as STALE_SOURCE.
8. If the trace does not contain enough information to verify a citation, mark it as unverifiable and explain why in unverifiable_notes.
9. Assign severity: CRITICAL for safety, legal, or compliance-impacting misalignments; MAJOR for factual errors with user impact; MINOR for imprecise but not misleading attributions.
10. Do not invent source content. Only use the provided [SOURCE_DOCUMENTS].

## RISK LEVEL
[RISK_LEVEL]

## EXAMPLES
[EXAMPLES]

To adapt this template, replace each square-bracket placeholder with data from your production trace system. The [TRACE_EVENTS] placeholder expects a JSON array of trace spans, including retrieval calls, reranking steps, and generation events with their associated document IDs. The [SOURCE_DOCUMENTS] placeholder should contain the full text of every document referenced in the trace, keyed by the same document IDs. The [RISK_LEVEL] placeholder accepts values like HIGH, MEDIUM, or LOW and can be used to adjust the strictness of severity classification. The [EXAMPLES] placeholder should be populated with a few representative citation alignment examples, including both aligned and misaligned cases, to calibrate the model's judgment. For production use, always validate the output JSON against the schema before ingesting it into dashboards or alerting systems. If the workflow involves regulated content, route CRITICAL severity findings to a human reviewer before taking automated action.

This prompt is designed to run as a post-hoc audit step, not as part of the real-time generation path. It consumes a structured trace object containing the final generated answer, its inline citations, and the full set of retrieved source documents with their trace event IDs. The ideal integration point is within an asynchronous evaluation pipeline triggered after a user session completes, a batch of traces is collected, or a compliance review is initiated. Do not run this prompt synchronously before returning an answer to the user; the latency and cost are justified only for offline quality assurance, regulatory review, or systematic hallucination monitoring.

To wire this into an application, build a trace pre-processor that extracts the required fields from your observability store (e.g., LangSmith, Arize, or a custom trace database) and assembles them into the [TRACE_INPUT] placeholder. The input must include: the full generated text with citation markers in situ, a mapping of citation IDs to source document identifiers, and the complete text of each retrieved source document. The model choice should prioritize strong reasoning and structured output capabilities—GPT-4o or Claude 3.5 Sonnet are appropriate defaults. Set temperature=0 for deterministic scoring. Implement a retry wrapper with up to 3 attempts for malformed JSON responses, using a repair prompt from the Output Repair and Validation Prompts pillar if the initial parse fails. Log every audit result with the trace ID, timestamp, model version, and prompt version to maintain an audit trail for governance reviews.

After receiving the alignment scorecard, validate the output programmatically before accepting it. Confirm that every misalignment instance references a real trace event ID present in the input, that the overall alignment score is a float between 0.0 and 1.0, and that the number of cited claims matches the number of evaluated claims. If the score falls below a defined threshold (e.g., 0.95 for regulated use cases), route the trace to a human review queue with the misalignment instances pre-populated. For high-volume pipelines, consider sampling traces by risk tier rather than auditing 100% of traffic. Avoid the temptation to use this prompt as a real-time guardrail—its strength is in systematic, evidence-backed audit, not in blocking responses under latency pressure.