Multi-Model Citation Accuracy Regression Prompt Template

This playbook is for RAG system owners and AI infrastructure teams who need to verify that generated answers remain faithfully grounded in provided source documents across different models. When you change foundation models, upgrade a model version, or switch providers, citation behavior can silently degrade. A model might start attributing claims to the wrong source, invent plausible-sounding references, or drop citations entirely. This prompt template standardizes the evaluation of citation accuracy into a repeatable regression test. It takes a question, a set of retrieved passages, and a generated answer with citations, then produces a structured citation faithfulness report per model.

Use this prompt before any model migration, provider switch, or RAG pipeline change that could affect grounded generation quality. It is designed for teams that already have a working RAG system and need to quantify whether a new model maintains or degrades citation discipline. The prompt works best when you have a curated set of test cases—each containing a question, retrieved passages, and a reference answer with known-good citations—that you can run across your current and candidate models. Do not use this prompt for evaluating retrieval quality itself; it assumes the retrieved passages are already provided and only assesses whether the model's citations faithfully map to those passages. For retrieval evaluation, use a separate retrieval quality prompt from the RAG pillar.

The output is a structured report with a per-claim faithfulness analysis, a citation accuracy score, and categorized error types: unsupported attribution, hallucinated source, missing citation, and correct citation. Wire this into your CI/CD pipeline as a regression gate. If the citation faithfulness score drops below your threshold after a model change, block the migration until you investigate. For high-stakes domains like legal, medical, or financial applications, always include human review of flagged citation errors before accepting the report as final. The prompt includes a [RISK_LEVEL] parameter that adjusts the strictness of the evaluation and whether borderline cases are flagged for human review.

This template provides the core instruction set for a multi-model citation accuracy regression test. It is designed to be executed against a golden dataset of question-answer pairs, each grounded in a set of provided source passages. The prompt instructs the model to act as an evaluator, comparing a generated answer against the sources to identify unsupported claims, hallucinated citations, and missing attributions. The output is a structured JSON report, making it suitable for automated comparison across different models or model versions.

textCopy
You are a citation faithfulness evaluator. Your task is to compare a generated answer against the provided source passages and assess the accuracy of its citations.

## INPUT
**Question:** [QUESTION]
**Generated Answer:** [GENERATED_ANSWER]
**Source Passages:**
[SOURCE_PASSAGES]

## EVALUATION CRITERIA
For each claim in the Generated Answer that references a source, determine if the claim is fully supported, partially supported, or contradicted by the cited passage. Also, flag any factual claims that lack a citation and any citations that reference a non-existent source.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "citation_faithfulness_score": <float 0.0 to 1.0>,
  "claims_analysis": [
    {
      "claim_text": "<string>",
      "cited_source_id": "<string or null>",
      "verdict": "<supported|partially_supported|contradicted|no_citation|hallucinated_source>",
      "explanation": "<string>"
    }
  ],
  "overall_assessment": "<string>"
}

## CONSTRAINTS
- Base your evaluation strictly on the provided Source Passages. Do not use outside knowledge.
- If a claim is a direct quote, it must be an exact match to the source text to be considered 'supported'.
- A 'hallucinated_source' verdict means the cited source ID does not exist in the provided passages.
- The 'citation_faithfulness_score' should be the proportion of supported claims out of all claims that should have a citation.

To adapt this template, replace the placeholders with your test data. [QUESTION] and [GENERATED_ANSWER] form the test case, while [SOURCE_PASSAGES] should be a clearly delimited list of text blocks, each with a unique ID. For automated regression testing, iterate this prompt over a golden dataset and execute it against each target model. Compare the citation_faithfulness_score and the distribution of verdicts across models to detect regressions. A drop in the score or an increase in hallucinated_source verdicts on a new model version is a clear failure signal that should block a release.

This prompt is designed to run inside an automated regression harness, not as a one-off manual check. The typical integration pattern is a Python or TypeScript evaluation runner that iterates over a golden dataset of RAG responses, sends each test case to multiple target models, collects the structured citation accuracy reports, and writes results to a database or CSV for analysis. Each test case should include the original user query, the full retrieved context provided to the model, and the model's generated answer with its inline citations. The harness must track metadata for every run: model identifier, model version, provider, timestamp, prompt template version, and the raw input/output payloads for auditability.

Validation is critical because the prompt's output is a structured JSON report. Implement a post-response validation layer that checks: (1) the response is valid JSON, (2) the citations array contains only citation IDs that exist in the provided context, (3) the claims array is non-empty and each claim references only valid citation IDs, (4) the citation_faithfulness_score is a float between 0.0 and 1.0, and (5) the unsupported_claims and hallucinated_sources arrays are present even if empty. If validation fails, retry the prompt once with the validation error message appended as a [CORRECTION_FEEDBACK] field. If the retry also fails validation, flag the test case as VALIDATION_FAILED and log the raw response for manual inspection. Do not silently accept malformed outputs as zero scores—this masks real model failures.

For model selection, run this prompt against every model in your deployment matrix. At minimum, include your current production model, any candidate upgrade models, and at least one baseline model for comparison. Use the same prompt template, same temperature (recommend 0.0 for reproducibility), and same context window across all models. If a model has a shorter context window, truncate the retrieved context consistently using a documented truncation strategy rather than silently dropping passages, which would invalidate cross-model comparisons. Log any truncation events in the run metadata.

The harness should produce a per-model citation faithfulness score aggregated across the full test suite, plus per-claim accuracy rates. Store results in a structured format that supports trend analysis: model, test_case_id, faithfulness_score, claim_count, supported_claim_count, unsupported_claim_count, hallucinated_source_count, and validation_status. Use these metrics to set regression gates—for example, a model upgrade must not decrease the aggregate faithfulness score by more than 2 percentage points and must not introduce new hallucinated source patterns on previously clean test cases. Wire these gates into your CI/CD pipeline so that model migrations are blocked if citation accuracy regresses.

Human review is required for any test case where the LLM judge's faithfulness assessment conflicts with a known ground-truth label in your golden dataset. If your dataset includes human-verified citation accuracy annotations, automatically flag cases where the model-assigned score deviates from the human label by more than 0.3. These discrepancies often reveal either annotation errors in your dataset or systematic blind spots in the evaluation prompt itself. Schedule a recurring review of flagged cases to improve both the golden dataset and the evaluation rubric over time.