Groundedness Rubric for Summarization Outputs

This prompt is a judge prompt, not a generation prompt. Its sole job is to evaluate an already-produced summary against its source document using a multi-dimensional rubric. The primary job-to-be-done is automated quality monitoring: you need a repeatable, scalable way to score how faithfully a summary represents the original text without manually reviewing every output. The ideal user is an AI engineer or evaluation lead who is building a batch evaluation pipeline, a regression testing suite for summarization models, or a pre-release gate that blocks deployment if groundedness scores fall below a defined threshold. You should use this prompt when you have a source document and a candidate summary, and you need a structured scorecard that breaks down factual consistency, omission of key information, and insertion of unsupported content.

This prompt is designed for batch evaluation workflows, not real-time user-facing features. Wire it into a pipeline that processes hundreds or thousands of document-summary pairs, producing structured scores you can aggregate, track over time, and use for model selection or drift detection. The output is a scorecard—typically JSON with dimension-level scores, specific violation flags, and a summary-level groundedness rating—that feeds into your monitoring dashboards or CI/CD checks. You must provide the full source document and the complete summary as inputs; this prompt cannot evaluate partial or streaming outputs reliably. The rubric dimensions (factual consistency, omission, insertion) are fixed in the template, but you can adjust the scoring scale and severity thresholds to match your team's risk tolerance.

Do not use this prompt when you need to generate a summary, when you lack the full source document, or when the summary is intended to be abstractive rather than extractive without clear grounding expectations. This prompt is also inappropriate for evaluating creative or opinion-based summaries where factual precision is not the primary quality dimension. If your use case involves regulated content (e.g., clinical summaries, legal document abstracts, financial report digests), you must pair this automated evaluation with human review for any outputs flagged below your acceptance threshold. The rubric provides a signal, not a final verdict. After reading this section, proceed to the prompt template to copy and adapt the judge instructions, then review the implementation harness for validation, retry logic, and logging patterns.

This prompt template operationalizes a structured evaluation of how faithfully a summary represents its source document. It moves beyond a simple 'good/bad' judgment by scoring five distinct dimensions: factual consistency, key information omission, unsupported content insertion, entity and relationship preservation, and causal/logical structure. The output is a strict JSON scorecard designed for direct ingestion into evaluation pipelines, dashboards, or CI/CD gates.

textCopy
You are an expert evaluator of summarization faithfulness. Your task is to score a summary against its source document using a multi-dimensional groundedness rubric. You will assess factual consistency, omission of key information, and insertion of unsupported content. Return a structured JSON scorecard with per-dimension scores, specific examples of violations, and an overall faithfulness rating.

### SOURCE DOCUMENT

[SOURCE_DOCUMENT]

codeCopy

### SUMMARY TO EVALUATE

[SUMMARY]

codeCopy

### EVALUATION RUBRIC

Score each dimension on a 1-5 scale where:
- 5: Perfect. No issues detected.
- 4: Minor issues that do not change meaning.
- 3: Some issues that partially affect accuracy.
- 2: Significant issues that undermine trust in the summary.
- 1: Severe issues. Summary is unfaithful to the source.

**Dimension 1: Factual Consistency**
Do all factual claims in the summary match the source document? Penalize contradictions, altered numbers, changed entities, or reversed relationships.

**Dimension 2: Key Information Omission**
Does the summary omit critical facts, events, or conclusions that a reader would need to understand the source? Penalize missing essential information even if what is present is accurate.

**Dimension 3: Unsupported Content Insertion**
Does the summary introduce claims, opinions, interpretations, or details not present in the source? Penalize any added content regardless of whether it seems plausible.

**Dimension 4: Entity and Relationship Preservation**
Are people, organizations, locations, dates, and their relationships preserved correctly? Penalize swapped entities, merged identities, or altered relationships.

**Dimension 5: Causal and Logical Structure**
Does the summary preserve the source's causal chain, argument structure, and logical ordering? Penalize reversed causality, broken if-then relationships, or reordered events that change meaning.

### OUTPUT FORMAT
Return ONLY valid JSON with this exact structure:
```json
{
  "overall_faithfulness_score": <float 1.0-5.0>,
  "overall_rating": "<Excellent|Good|Fair|Poor|Unacceptable>",
  "dimensions": {
    "factual_consistency": {
      "score": <int 1-5>,
      "explanation": "<brief reasoning>",
      "violations": ["<specific example from summary with source comparison>"]
    },
    "key_information_omission": {
      "score": <int 1-5>,
      "explanation": "<brief reasoning>",
      "omitted_items": ["<key fact present in source but missing from summary>"]
    },
    "unsupported_content_insertion": {
      "score": <int 1-5>,
      "explanation": "<brief reasoning>",
      "inserted_items": ["<claim in summary with no source support>"]
    },
    "entity_relationship_preservation": {
      "score": <int 1-5>,
      "explanation": "<brief reasoning>",
      "violations": ["<specific entity or relationship error>"]
    },
    "causal_logical_structure": {
      "score": <int 1-5>,
      "explanation": "<brief reasoning>",
      "violations": ["<specific causal or logical error>"]
    }
  },
  "critical_failures": ["<any violation that alone would make the summary unacceptable>"],
  "evaluation_notes": "<optional overall assessment>"
}

CONSTRAINTS

• Every violation must quote or reference specific text from both the summary and the source.
• If no violations exist for a dimension, return an empty array.
• The overall_faithfulness_score is the mean of the five dimension scores.
• Map overall_faithfulness_score to rating: >=4.5 Excellent, >=3.5 Good, >=2.5 Fair, >=1.5 Poor, <1.5 Unacceptable.
• Do not evaluate writing style, grammar, or conciseness. Only evaluate factual groundedness.
• If the source document is empty or the summary is empty, return an error object: {"error": "<reason>"}.

[OPTIONAL_KEY_FACTS]

INSTRUCTIONS

Evaluate the summary now. Return only the JSON object with no additional text.

To adapt this template, replace [SOURCE_DOCUMENT] and [SUMMARY] with your actual text inputs. The [OPTIONAL_KEY_FACTS] placeholder can be used to inject a pre-extracted list of critical facts that must appear in a faithful summary, which helps calibrate the omission dimension for domain-specific requirements. For high-stakes domains like clinical documentation or legal review, always route outputs scoring below 'Good' (3.5) to a human review queue before any downstream action is taken.

Before deploying this prompt into a production evaluation pipeline, validate the JSON output against the exact schema using a structural validator. Common failure modes include the model returning markdown-wrapped JSON, omitting the critical_failures array, or producing scores that don't match the documented rating thresholds. Implement a retry with a stricter format reminder if parsing fails, and log all raw outputs for judge calibration analysis.

This prompt is designed for offline batch evaluation, not real-time user flows. It should be called via your LLM API with response_format set to JSON mode if the provider supports it. The returned JSON must be parsed and validated before any score is trusted. The core validation contract is: all five dimension scores (factual_consistency, omission_rate, insertion_rate, source_fidelity, claim_support_ratio) must be integers between 1 and 5, and the overall_faithfulness_score must equal the arithmetic mean of those five values. If validation fails, retry once with the identical prompt payload. If the retry also fails, log the raw response alongside the input summary and source document identifiers, then flag the record for human review. Do not silently accept malformed scores.

Store dimension scores as time-series metrics tagged by model version, prompt version, and evaluation batch ID. This enables drift detection: if insertion_rate scores trend upward across batches, your summarization model may be hallucinating more over time. For high-stakes domains such as legal document summarization or clinical note synthesis, route any summary scoring below 3.0 overall to a human review queue before the output reaches a downstream system or user. The rubric prompt is not a streaming guardrail—it is too latent and too coarse for per-token intervention. Accept that latency is tolerable here because this runs asynchronously after generation.

Model choice matters. Use a model with strong instruction-following and JSON output reliability for the judge role. If your summarization model is a smaller, fine-tuned model, the judge should typically be a more capable model to reduce judge bias. Avoid using the same model instance as both generator and judge without calibration against human ratings. Wire the evaluation results into your existing observability stack—log failures, score distributions, and validation errors alongside generation traces so you can correlate low groundedness scores with specific prompt or retrieval changes. The next step after implementing this harness is to calibrate the judge against a human-annotated sample to confirm score alignment before relying on it for automated gating decisions.