Citation Compliance Monitoring Prompt Template

This prompt is designed for RAG platform engineers and quality assurance teams who need to programmatically verify that every claim in a production response is backed by a specific, retrievable source document. The core job-to-be-done is not just to check if citations exist, but to audit their coverage (are all factual claims cited?) and accuracy (does the cited source actually support the claim?). The ideal user is integrating this prompt into a continuous monitoring pipeline where it acts as an automated compliance auditor, processing a stream of production traces that include the final user-facing response, the list of cited document IDs, and the full text of those retrieved documents.

Use this prompt when you have access to the full trace context—specifically the generated_response, retrieved_documents array, and citation_map. It is a post-hoc analysis step, not a real-time generation guardrail. It is most effective when your RAG system already attempts to produce citations and you need to measure its reliability against a defined compliance standard (e.g., '100% of declarative factual statements must have a citation with a confidence score > 0.9'). Do not use this prompt for real-time response filtering due to its inherent latency and cost; instead, use its structured output to trigger asynchronous alerts or to populate a compliance dashboard. It is also unsuitable for evaluating subjective opinions, stylistic choices, or creative text generation where factual grounding is not a strict requirement.

Before deploying this prompt, you must define a clear [COMPLIANCE_POLICY] that specifies what constitutes a violation. A vague policy like 'cite sources' will produce noisy, unactionable results. A strong policy defines specific schemas: for example, 'A citation is non-compliant if the cited text does not contain the key entities from the claim' or 'A missing citation is triggered for any sentence containing a numerical statistic, a named entity, or a technical specification.' The prompt's output is a structured compliance report, not a simple pass/fail boolean. It produces a compliance_score, a list of missing_citations, and a list of inaccurate_citations with trace-level correlation IDs. This detailed output is what allows you to build reliable alert thresholds, such as 'Alert if compliance_score < 0.95 over a 10-minute rolling window.' The next step after reading this section is to prepare your trace data schema and define your organization's specific citation compliance rules before copying the prompt template.

This prompt template is designed to be integrated into a continuous monitoring pipeline. It takes a single production trace—including the user query, retrieved context, final response, and any existing citations—and evaluates it against a configurable set of compliance rules. The output is a structured JSON object containing a compliance score, a list of specific violations, and source-level attribution gaps, ready for ingestion by alerting systems.

textCopy
You are a citation compliance auditor for a production RAG system. Your task is to evaluate a single AI-generated response against a set of citation rules and the provided source documents.

## INPUT

**User Query:**
[USER_QUERY]

**Retrieved Context (source_id: content):**
[RETRIEVED_CONTEXT]

**Generated Response:**
[GENERATED_RESPONSE]

**Citation Schema:**
Citations in the response are formatted as markdown links: `[source_id]`.

## COMPLIANCE RULES

[COMPLIANCE_RULES]

## INSTRUCTIONS

1.  **Extract Claims:** Identify every factual claim in the Generated Response.
2.  **Verify Citations:** For each claim, check if it is supported by a citation and if that citation points to a valid `source_id` in the Retrieved Context.
3.  **Verify Accuracy:** For each cited claim, verify that the content of the cited source accurately and directly supports the claim. Flag any misrepresentation or hallucination.
4.  **Check for Gaps:** Identify any factual claims that are missing a required citation.
5.  **Check for Orphans:** Identify any citations in the response that point to `source_id`s not present in the Retrieved Context.

## OUTPUT_SCHEMA

Return your analysis as a single, valid JSON object conforming to this schema:
{
  "compliance_score": <number 0.0 to 1.0>,
  "verdict": "compliant" | "non_compliant",
  "violations": [
    {
      "type": "unsupported_claim" | "missing_citation" | "orphan_citation" | "inaccurate_citation",
      "claim_text": "<the exact claim text>",
      "source_id": "<the cited source_id or null>",
      "reason": "<brief explanation of the violation>"
    }
  ],
  "source_attribution_gaps": [
    {
      "source_id": "<source_id>",
      "issue": "<description of why this source is a gap, e.g., 'cited but not in context', 'in context but never cited'>"
    }
  ]
}

## CONSTRAINTS

- Base your analysis strictly on the provided User Query, Retrieved Context, and Generated Response.
- Do not use outside knowledge.
- If no violations are found, the `violations` and `source_attribution_gaps` arrays must be empty.
- The `compliance_score` must be 1.0 if and only if there are zero violations.

To adapt this template, replace the [COMPLIANCE_RULES] placeholder with your organization's specific citation policy, such as 'Every factual claim must be cited' or 'Claims derived from source X require a direct quote.' The [RETRIEVED_CONTEXT] should be formatted as a clear mapping of source IDs to their text content. For high-stakes domains, ensure that a human review step is triggered for any non_compliant verdict before the response is surfaced to an end-user or logged as a systemic failure.

The Citation Compliance Monitoring prompt is designed to run as a post-generation evaluation step, not as part of the primary user-facing response flow. After your RAG system produces an answer with citations, pass the generated answer, the retrieved source documents, and the expected citation schema into this prompt. The model returns a structured compliance report that your application can parse and act on. This separation keeps compliance checking out of the latency-critical generation path and allows you to run it asynchronously or in batch.

Wire the prompt into an evaluation harness that calls your LLM provider's API with response_format set to json_object (or the equivalent structured output mode for your model). The output schema should be validated immediately after the API call returns. At minimum, validate that compliance_score is a float between 0.0 and 1.0, that missing_citations is an array of objects with claim_text and source_document_id fields, and that source_attribution_gaps contains document-level trace correlation identifiers. If validation fails, retry once with the validation errors appended to the prompt as additional [CONSTRAINTS]. After two consecutive validation failures, log the raw output and escalate for human review rather than silently accepting malformed compliance data.

For production observability, attach the compliance report to the original trace span using your observability platform's SDK (OpenTelemetry, Datadog, or custom logging). Tag the span with compliance_score, citation_count, and missing_citation_count as span attributes so they become queryable in dashboards and alert conditions. This enables the continuous monitoring workflows described in sibling playbooks like Trace-Based SLO Threshold Evaluation and Hallucination Rate Monitoring Query. If the compliance score drops below your defined threshold (typically 0.85 for high-trust domains), trigger an alert that includes the trace ID and affected document IDs for rapid root-cause investigation.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid smaller or older models that may hallucinate compliance scores or fail to correlate claims back to specific document chunks. If cost is a concern, consider running this prompt on a sampled percentage of production traffic (e.g., 10% for low-risk queries, 100% for regulated domains) rather than degrading accuracy with a cheaper model. Never use the same model instance that generated the original answer to also evaluate its own citation compliance; this creates a self-review blind spot that masks systematic citation failures.