PR Diff Changelog Generation Prompt

The raw unified diff or git diff output for the pull request. This is the primary source material for changelog generation.

diff --git a/src/auth/login.ts b/src/auth/login.ts @@ -12,6 +12,8 @@ ...

Must be non-empty and contain at least one file change. Validate with a diff parser or check that the string starts with 'diff --git' or contains '@@' hunk headers. Reject if the input is a commit message or prose summary instead of a diff.

The title of the pull request as written by the author. Used to infer intent and provide a human-readable anchor for the changelog entry.

fix: resolve session timeout during token refresh

Must be a non-empty string under 200 characters. If the title follows conventional commit format (type: description), the type prefix can be used to seed the change category. Null allowed if the title is unavailable; the prompt should fall back to diff analysis.

The body of the pull request description. Provides context, motivation, and manual notes that supplement the diff.

Users reported being logged out after 15 minutes of inactivity. This PR extends the refresh window and adds a keep-alive ping.

Optional string. If provided, should be under 5000 characters to avoid diluting the diff signal. Validate that the description is not a copy of the diff itself. Null allowed.

Specifies who the changelog is written for. Controls tone, technical depth, and whether internal implementation details are included.

end-users

Must be one of: 'end-users', 'developers', 'internal-team', 'enterprise-admins'. Default to 'developers' if not specified. Validate against the allowed enum before prompt assembly. Incorrect audience selection causes inappropriate technical depth.

The version number or release tag this changelog entry belongs to. Appended to the output for traceability.

v2.4.1

Must match a semantic versioning pattern (MAJOR.MINOR.PATCH) or a valid release tag string. Validate with a semver regex. Null allowed if the version is unknown at generation time; the prompt should omit the version line.

The most recent changelog entries for the project. Used to avoid duplicate entries and maintain consistent style and categorization.

v2.4.0

• Added: bulk user import via CSV
• Fixed: pagination offset on search results

Optional string. If provided, validate that it contains at least one changelog entry with a recognized category prefix (Added, Fixed, Changed, Deprecated, Removed, Security). Used for deduplication and style matching, not for content generation.

A brief description of the repository's purpose and primary domain. Helps the model correctly interpret domain-specific changes.

Backend API for a healthcare scheduling platform handling patient appointments and provider availability.

Must be a non-empty string under 500 characters. Should describe what the software does, not how it's built. Validate that the context is factual and does not contain prompt injection or instruction-like language.

Specifies the structure of the generated changelog. Controls whether the output is Markdown, JSON, or plain text.

markdown

Must be one of: 'markdown', 'json', 'plaintext'. Default to 'markdown'. Validate against the allowed enum. If 'json' is selected, the prompt must include a JSON schema for the output and the harness must validate the response against that schema.

object

Top-level object must contain exactly the fields listed in this contract; reject extra fields.

string (<= 120 chars)

Must be a human-readable summary of the change; parse check for imperative mood and absence of commit hashes.

enum: feature, fix, deprecation, security, performance, dependency, breaking_change, other

Must match one of the allowed enum values; reject unknown categories.

string (<= 500 chars)

Must explain the user-facing impact in plain language; must not contain raw code blocks, stack traces, or internal JIRA ticket IDs.

boolean

Must be true if category is breaking_change or if description indicates backward incompatibility; schema check for boolean type.

string or null

Required if breaking is true; must provide actionable upgrade steps for users. If breaking is false, must be null.

array of strings

Each element must match the pattern ^[a-zA-Z0-9_-.]+$ (PR identifier); reject URLs or full branch names.

enum: high, medium, low

Must be present; if low, the entry must be flagged for human review before publication.

Every entry is assigned exactly one correct category from [VALID_CATEGORIES] (e.g., feature, fix, deprecation).

A technical commit like 'refactor auth' is labeled as a user-facing 'feature'. A bug fix is labeled 'deprecation'.

Spot-check 20 entries against a pre-labeled golden set; require >=95% exact match accuracy.

A non-technical user can understand the value of each entry without reading the code. No internal jargon (e.g., 'fixed N+1 query').

Entry contains implementation details like 'updated library X to v2.3' or 'refactored the caching layer'.

Have 3 non-engineers rate each entry on a 1-5 clarity scale; require an average score of >=4.0 across all entries.

All entries categorized as 'deprecation' or 'breaking' include a non-empty [MIGRATION_NOTE] field.

A breaking change is listed without any guidance for the user on how to adapt.

Parse the JSON output; assert that for every entry where category == 'breaking', the 'migration_note' field is a non-null string with length > 20 characters.

Every generated entry must be directly attributable to a change present in the [PR_DIFF]. No invented features or fixes.

The changelog mentions a 'dark mode' feature that does not exist in the code diff.

Use an LLM-as-Judge with the diff and the output; prompt: 'Does the changelog contain any statement not supported by the diff?'. Require a 'No' verdict.

The output is a single valid JSON object matching the [OUTPUT_SCHEMA] exactly, with no extra keys or markdown fences.

Output is wrapped in json ... fences, or a required key like 'entries' is missing.

Automated JSON schema validator in the test harness; test must pass with zero validation errors.

Each entry description is a single, clear sentence under 150 characters.

An entry is a multi-sentence paragraph explaining the technical background of a minor fix.

Automated character count check on each entry's 'description' field; require all entries to be <= 150 characters.

Deprecation entries clearly state the replacement feature or recommended action, not just that something was removed.

Entry reads 'Removed the /v1 API endpoint' with no pointer to the /v2 replacement.

Parse entries where category == 'deprecation'; assert that the description contains a verb like 'use', 'migrate to', or 'replaced by'.

A low-confidence score (<0.7) correlates with genuinely ambiguous or massive diffs, not simple, clear changes.

A single-line typo fix is output with a confidence score of 0.5.

Run the prompt on 10 simple, single-line fix diffs; assert that all returned confidence scores are >= 0.85.