Example-Based Prompt QA Report Prompt

Raw pass/fail/tie outputs from the regression test harness, including per-example verdicts and diff summaries

{"test_case_id": "tc-042", "verdict": "fail", "diff_summary": "Missing required field 'disclaimer'", "old_output": "...", "new_output": "..."}

Must be valid JSON array of objects. Each object requires test_case_id, verdict, and diff_summary fields. Reject if verdict values are not strictly pass, fail, or tie.

Coverage gap report mapping tested vs. untested input dimensions, intent categories, and difficulty tiers

{"covered_dimensions": ["intent", "length"], "gaps": [{"dimension": "language", "severity": "high", "missing_categories": ["Spanish", "Japanese"]}]}

Must include both covered_dimensions and gaps arrays. Gaps with severity 'high' must trigger a risk item in the final report. Null allowed if coverage analysis was not performed.

Results from edge-case and boundary tests, including null inputs, maximum lengths, and conflicting constraints

[{"test_case_id": "edge-017", "input_type": "null_field", "verdict": "pass", "notes": "Correctly returned empty array"}]

Must be valid JSON array. Each object requires test_case_id, input_type, and verdict. Edge cases with verdict 'fail' must appear in the risk items section of the report.

Per-field schema compliance check results for structured outputs across the test suite

{"total_checks": 150, "failures": 3, "violations": [{"field": "confidence_score", "expected_type": "number", "actual_type": "string", "test_case_id": "tc-089"}]}

Must include total_checks, failures count, and violations array. Each violation requires field, expected_type, actual_type, and test_case_id. Null allowed if no structured output tests were run.

Clustered failure taxonomy from test runs with representative examples and frequency counts

[{"failure_mode": "schema_drift", "frequency": 12, "representative_example": "tc-034", "description": "Output missing newly required 'audit_trail' field"}]

Must be valid JSON array. Each entry requires failure_mode, frequency, and representative_example. Frequency counts must be non-negative integers. Null allowed if no failures occurred.

Human-readable diff or changelog between the previous prompt version and the candidate version under review

"Changed system prompt: added refusal policy for medical advice. Updated few-shot example 3 to include disclaimer field."

Must be a non-empty string. If no changes were made, use explicit value 'No changes from previous version'. Reject empty or whitespace-only strings.

Metadata about the release candidate, including target environment, model version, and deployment urgency

{"candidate_version": "v2.3.1", "target_model": "claude-sonnet-4-20250514", "environment": "staging", "urgency": "standard", "rollback_plan": "Revert to v2.3.0 via feature flag"}

Must include candidate_version, target_model, and environment. Urgency must be one of: standard, elevated, critical. Rollback_plan is required when urgency is critical. Reject if target_model does not match a known model identifier.

Pre-identified outputs or test cases that require mandatory human review before release approval

[{"test_case_id": "tc-101", "reason": "Output contains legal interpretation without disclaimer", "reviewer": "legal-team@example.com"}]

Must be valid JSON array. Each entry requires test_case_id, reason, and reviewer. If no human review is required, use empty array []. Do not use null.

string (UUID v4)

Must match UUID v4 regex. Auto-generated if missing.

string (ISO 8601 UTC)

Must parse as valid ISO 8601 datetime in UTC. Reject if future-dated beyond 5 minutes.

string

Must match the [PROMPT_VERSION] input exactly. Fail if mismatch.

string

Must match the [TEST_SUITE_NAME] input. Non-empty, max 255 characters.

integer

Must equal count of entries in results array. Fail on mismatch.

integer

Must be <= summary.total_tests. Sum of passed + failed + errors must equal total_tests.

integer

Must be >= 0. Each failed entry requires a corresponding failure_detail in results.

integer

Must be >= 0. Each error entry requires a non-null error_message in results.

number (float, 0.0-1.0)

Must equal passed / total_tests rounded to 4 decimal places. Fail if computed value deviates > 0.0001.

array of objects

Must be a non-empty array. Each element must conform to the result_item schema below.

string

Must match a test_case_id from the [GOLDEN_TEST_SET] input. Fail on unknown IDs.

enum: passed | failed | error

Must be one of the three allowed values. No other strings permitted.

object or string

Must match the expected_output from the corresponding golden test case. Schema check against golden.

object or string

Must be present. Null allowed only if status is error.

string or null

Required when status is failed. Must be non-empty string describing the regression. Null when status is passed.

string or null

Required when status is error. Must describe the execution error. Null when status is passed or failed.

string or null

Required when status is failed. Must summarize key differences between expected and actual. Null otherwise.

array of objects

May be empty array. Each item requires severity, description, and affected_test_ids fields.

enum: low | medium | high | critical

Must be one of the four allowed values. Fail on unrecognized severity.

string

Non-empty. Max 500 characters. Must reference specific failure patterns, not generic statements.

array of strings

Must contain at least one test_case_id present in results. Fail if any ID is not in the results array.

enum: go | no_go | conditional_go

Must be one of the three allowed values. conditional_go requires non-empty conditions field.

array of strings or null

Required when release_recommendation is conditional_go. Each string must describe a specific, verifiable condition. Null for go or no_go.

array of strings

May be empty. Each string must describe a specific input category or edge case not covered by the test suite.

Output is valid JSON matching the [OUTPUT_SCHEMA] with all required fields present

JSON parse error or missing required field

Schema validator with strict mode; check for null in required fields

Reported pass/fail counts match the [REGRESSION_RESULTS] input within ±1 tolerance

Mismatch between reported and actual pass/fail counts

Parse [REGRESSION_RESULTS] counts and compare to report summary fields

Every risk item references at least one specific test case ID or failure mode from [REGRESSION_RESULTS]

Risk item with no test case reference or generic claim without evidence

Regex check for test case ID pattern in each risk item; flag items without a match

Recommendation aligns with pass rate thresholds defined in [RELEASE_CRITERIA]: recommend 'go' only when pass rate meets threshold

Recommendation contradicts the pass rate or ignores [RELEASE_CRITERIA] thresholds

Extract pass rate and recommendation; assert logical consistency with threshold rules

Report includes a dedicated edge-case section with counts and pass/fail breakdown from [EDGE_CASE_RESULTS]

Edge-case section missing or counts don't match [EDGE_CASE_RESULTS] totals

Check for edge-case section header; compare reported counts to input totals

All schema violations from [SCHEMA_CHECK_RESULTS] are listed with field paths and violation types

Schema violation omitted or field path missing from report

Diff the set of violations in input against violations listed in report; flag missing entries

Report uses calibrated language for borderline results: 'likely', 'may indicate', 'requires review' for cases near thresholds

Overconfident language on borderline results or missing uncertainty qualifiers

LLM-as-judge check on a sample of borderline cases; flag absolute claims without hedging

Report flags any test case where semantic match score falls below [REVIEW_THRESHOLD] for human review

Test case below threshold not flagged or flag present without case ID

Extract all flagged case IDs; verify each has a score below [REVIEW_THRESHOLD] in input data