Release Note Hallucination and Factuality Check Prompt

This prompt is designed for release managers, DevRel engineers, and platform teams who need to audit AI-generated release notes for factual accuracy before they reach users. The core job-to-be-done is claim-by-claim verification: the model receives a draft release note and the corresponding source material (commit diffs, merged PR descriptions, and issue tracker references) and must produce a structured audit report. Each claim in the release note is mapped to supporting evidence, flagged as unsupported if no evidence exists, and assigned a confidence score. This is not a prompt for generating release notes—it is a verification gate that sits between generation and publication.

The ideal user has already run a generation step (such as the Commit Diff to Release Note Prompt Template) and now needs to prevent hallucinated features, incorrect version references, or misleading bug fix descriptions from reaching customers. Required inputs include the full draft release note text, the raw diff or commit log that the note claims to summarize, and any linked issue or PR metadata. The prompt expects these inputs to be provided in structured blocks so the model can perform line-level traceability. Without complete source material, the verification will produce low-confidence results and should not be trusted.

Do not use this prompt when the source material is incomplete, when the release note covers changes outside the provided diff window, or when the release note includes forward-looking statements (roadmap items, future deprecations) that cannot be verified against code changes. This prompt is also inappropriate for real-time or streaming release note generation pipelines where latency constraints prevent thorough audit steps. For high-risk releases—security patches, data migration releases, or compliance-mandated changes—always route the audit output through human review before publication. The prompt identifies unsupported claims but cannot independently decide whether an unsupported claim is a hallucination or a documentation gap that a human should fill.

This prompt template is the core of the release note factuality audit workflow. It instructs the model to act as an auditor that compares every claim in a draft release note against the source code diffs and commit messages that supposedly produced it. The template is designed to be dropped into an automated CI pipeline or a manual review queue. It forces claim-by-claim traceability, requires explicit evidence mapping, and produces a structured verdict that downstream systems can parse. The square-bracket placeholders let you inject the specific draft, the source material, and your risk tolerance without rewriting the auditing logic each time.

codeCopy
You are a release note factuality auditor. Your job is to verify whether every claim in a draft release note is supported by the provided source diffs and commit messages. You do not add, improve, or rewrite the release note. You only audit it.

## INPUTS
- Draft release note:
  [DRAFT_RELEASE_NOTE]
- Source diffs (unified diff format):
  [SOURCE_DIFFS]
- Commit messages (one per line, with SHA):
  [COMMIT_MESSAGES]
- Risk level: [RISK_LEVEL]
  Options: low | medium | high

## AUDIT RULES
1. Extract every factual claim from the draft release note. A factual claim is any statement that asserts a change was made, a feature was added, a bug was fixed, a dependency was updated, a breaking change was introduced, or a behavior was modified.
2. For each claim, search the source diffs and commit messages for supporting evidence.
3. If evidence is found, cite the specific file path, line range, or commit SHA that supports the claim.
4. If no evidence is found, flag the claim as UNSUPPORTED.
5. If partial evidence exists but the claim overstates the change, flag it as OVERSTATED and explain the gap.
6. If the draft release note omits a change that appears in the source diffs and meets the severity threshold for [RISK_LEVEL], flag it as MISSING.
7. Do not flag style-only changes, comment-only changes, or CI configuration changes unless [RISK_LEVEL] is high.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "audit_summary": {
    "total_claims": <int>,
    "supported_claims": <int>,
    "unsupported_claims": <int>,
    "overstated_claims": <int>,
    "missing_changes": <int>,
    "overall_confidence": <float between 0.0 and 1.0>
  },
  "claims": [
    {
      "claim_id": <int>,
      "claim_text": "<exact text from draft>",
      "verdict": "supported | unsupported | overstated",
      "evidence": [
        {
          "source_type": "diff | commit",
          "source_reference": "<file path:line range or commit SHA>",
          "explanation": "<how this evidence supports or fails to support the claim>"
        }
      ],
      "correction_suggestion": "<if unsupported or overstated, suggest corrected text; otherwise null>"
    }
  ],
  "missing_changes": [
    {
      "change_description": "<what changed>",
      "source_reference": "<file path:line range or commit SHA>",
      "severity": "breaking | feature | bugfix | dependency | other",
      "suggested_entry": "<how this should appear in the release note>"
    }
  ]
}

## CONSTRAINTS
- Do not invent evidence. If you cannot find a source, the verdict must be unsupported.
- Do not rephrase claims. Quote the draft release note exactly in claim_text.
- If [RISK_LEVEL] is high, flag even minor omissions. If low, only flag breaking changes and security fixes as missing.
- If the draft release note is empty or contains only placeholder text, return an audit with total_claims: 0 and overall_confidence: 0.0.

To adapt this template, start by adjusting the [RISK_LEVEL] thresholds to match your team's release policy. For security-sensitive products, you may want to add a [SECURITY_POLICY] placeholder that defines which CVE severities or vulnerability classes require mandatory inclusion. The output schema is designed to be machine-parseable, so wire the overall_confidence field into your release gate: if confidence drops below your threshold, block the release note from publication and route it for human review. For teams using multiple models, test this prompt across your candidate models with a golden dataset of known hallucinations to calibrate which model catches the most unsupported claims without generating false positives on correctly grounded statements.

This prompt is designed to run as a post-generation verification step, not as a standalone content creator. In a typical implementation, a release note draft is first generated from commit diffs or merged PRs using a separate generation prompt. The output of that generation step becomes the [DRAFT_RELEASE_NOTE] input for this factuality check prompt. The system should also supply the [SOURCE_DIFFS] or [COMMIT_LOG] that were used to produce the draft, along with any [PACKAGE_MANIFEST] or [CHANGELOG_HISTORY] for cross-referencing. The prompt expects these inputs to be assembled before invocation, and the model should never be asked to retrieve or infer source material on its own.

To wire this into an application, wrap the prompt in a function that accepts the draft text and source evidence as parameters. Before calling the model, validate that the source evidence is non-empty and that the draft text exceeds a minimum length threshold—an empty or trivially short draft should fail fast without consuming inference tokens. After receiving the model response, parse the output as JSON and validate the schema: each claim object must contain claim_text, source_evidence, confidence_score, and verdict fields. If the JSON is malformed, retry once with a stricter output format instruction appended to the prompt. Log every verification run with a unique run_id, the input hashes, and the full model response for auditability. For high-risk releases, route any claim with a confidence_score below 0.7 or a verdict of unsupported to a human review queue before the release note is published.

Model choice matters here. Use a model with strong instruction-following and JSON output capabilities, such as gpt-4o or claude-3-5-sonnet, and set the temperature low (0.0–0.2) to maximize factual consistency and schema adherence. Avoid using smaller or older models that are more prone to hallucinating source citations or misclassifying supported claims as unsupported. If your release pipeline processes many commits, consider chunking the source evidence and running the verification in parallel across multiple prompt calls, then merging the results. However, be aware that parallel runs may produce overlapping claims that require deduplication. Always include a final aggregation step that reconciles duplicate claims and flags any draft statements that were not covered by any verification chunk.

Do not treat this prompt as a replacement for human review in regulated or safety-critical software. If your release notes cover security patches, medical device software, or financial systems, the output of this prompt should be treated as an advisory check, not a final gate. Build an approval workflow where a human release manager reviews all flagged claims and confirms the final release note before publication. The prompt's confidence scores are useful for prioritization—sort the review queue by lowest confidence first—but they are not a substitute for domain expertise when the stakes are high.