Test Retirement Impact Assessment Prompt

List of test case IDs proposed for retirement with current execution metadata

TC-4021 (avg runtime 12s, last failure 2024-11-03), TC-4088 (avg runtime 45s, last failure 2024-09-17)

Must be a non-empty array of valid test IDs from the test management system. Reject if any ID is not found in the current test registry. Each entry must include runtime and last-failure-date fields.

Mapping from each test candidate to the requirements, code paths, or risk items it currently covers

TC-4021 covers REQ-AUTH-07 (OAuth token refresh), TC-4088 covers REQ-PAY-12 (payment retry logic) and REQ-PAY-14 (idempotency key validation)

Every candidate in [TEST_CANDIDATES] must appear as a key. Coverage entries must reference traceable requirement IDs or code path identifiers from the current coverage report. Reject if coverage data is older than the last release cycle.

Information about the upcoming release: scope, risk profile, regulatory constraints, and stakeholder list

Release v2.9.0: PCI-DSS scope, payment module refactor, 3 dependent services (billing, fraud, notifications). Stakeholders: payments-team, security-architect, qa-lead

Must include release version, regulatory tags if applicable, and at least one stakeholder group. Null allowed for regulatory tags if release has no compliance impact. Reject if release scope is empty or undefined.

Recent execution results for each candidate: pass/fail trends, flakiness score, defect detection record

TC-4021: 98% pass rate last 30 days, 0 defects found in 6 months. TC-4088: 72% pass rate, 3 unique defects caught in last quarter, flakiness score 0.4

Must cover the time window specified in [RELEASE_CONTEXT] or default to last 90 days. Flakiness score must be a float between 0.0 and 1.0. Reject if pass-rate data is missing for any candidate.

Downstream consumers of each test: CI pipelines, deployment gates, compliance reports, dependent test suites

TC-4021 runs in ci-staging-gate and compliance-audit-suite. TC-4088 runs in ci-production-gate and payment-regression-suite

Must list pipeline names or suite identifiers that execute each candidate. Reject if a candidate appears in a production deployment gate and no replacement coverage is proposed. Null allowed for candidates with no downstream consumers.

Existing or planned tests that cover the same risk areas as the retirement candidates

REQ-AUTH-07 also covered by TC-4100 (integration test). REQ-PAY-12 and REQ-PAY-14 partially covered by TC-4205 (contract test, missing idempotency edge case)

For each coverage entry in [COVERAGE_MAP], provide at least one replacement test ID or mark as UNCOVERED. Reject if a requirement marked as critical in [RELEASE_CONTEXT] has no replacement coverage and no explicit risk acceptance.

Hard rules for the assessment: minimum coverage thresholds, approval requirements, blocked retirement conditions

Cannot retire any test covering PCI-DSS requirements without security-architect approval. Minimum 85% coverage preservation for payment module. Block retirement if candidate detected a defect in last 60 days.

Each constraint must be a boolean-evaluable condition. Reject if constraints contradict each other. At least one constraint must address regulatory or compliance requirements if [RELEASE_CONTEXT] includes regulatory tags.

Expected structure of the impact report: per-candidate recommendation, coverage gap list, risk summary, stakeholder notifications, go/no-go decision

JSON schema with fields: candidate_id, recommendation (RETIRE|RETAIN|DEFER), coverage_gaps[], risk_level (LOW|MEDIUM|HIGH|CRITICAL), affected_stakeholders[], go_no_go (true|false), justification (string)

Schema must be provided as a valid JSON Schema or TypeScript interface. Reject if schema lacks required fields for audit trail. Every candidate in [TEST_CANDIDATES] must produce an entry in the output array.

string (UUID v4)

Must be a valid UUID v4 generated by the model. Reject if missing or malformed.

array of objects

Array must not be empty. Reject if null or length 0. Each object must conform to the candidate schema.

string

Must match an input test case ID from [CANDIDATE_IDS]. Reject if ID not in the submitted list.

enum: retire | retain | defer

Must be one of the three allowed values. Reject any other string.

enum: low | medium | high | critical

Must be one of the four allowed values. Reject if missing or invalid.

enum: low | medium | high | critical

Must be one of the four allowed values. Reject if missing or invalid.

integer >= 0

Must be a non-negative integer. Reject if negative or non-integer.

array of strings

Array must contain at least one stakeholder team name. Reject if empty or null.

string (max 500 chars)

Must be present and non-empty. Reject if length exceeds 500 characters.

array of strings

If recommendation is 'retire', this field must contain at least one test case ID that covers the same risk. Reject if missing when recommendation is 'retire'.

enum: go | no_go | conditional_go

Must be one of the three allowed values. Reject if missing or invalid.

string (max 1000 chars)

Must be present and non-empty. Reject if length exceeds 1000 characters.

string (ISO 8601 UTC)

Must be a valid ISO 8601 timestamp in UTC. Reject if unparseable or missing timezone.

=90% agreement with known labels on golden dataset

Prompt recommends retiring tests that should be kept or keeping tests that should be retired

Run prompt against 20-30 pre-labeled cases; compute precision, recall, F1 for retire vs keep decisions

Prompt identifies >=80% of known coverage gaps introduced by retirement candidates

Prompt claims no coverage gaps when known gaps exist, or invents gaps not present in the coverage map

Compare prompt-identified gaps against a pre-mapped coverage gap list for each retirement candidate

Prompt lists all 3 required stakeholder categories: test owners, downstream consumers, and release managers

Missing one or more stakeholder categories; generic placeholder like 'relevant teams' instead of specific roles

Parse output for stakeholder section; check presence of owner, consumer, and release manager categories

Risk justification references concrete code paths, features, or regression history for each candidate

Vague rationale like 'low risk' or 'not important' without linking to specific coverage or defect data

Human review of risk rationale text; check for presence of specific code path, feature name, or defect reference

Estimated time savings within 20% of actual execution time for each candidate

Savings estimate off by more than 50% or missing entirely for candidates with known execution duration

Compare prompt output savings estimate against pre-measured execution times from test run history

Output matches expected schema with all required fields present and correctly typed

Missing required fields, wrong types, or extra fields that break downstream parsing

Validate output against JSON schema; flag missing fields, type mismatches, or unexpected keys

Prompt correctly handles borderline cases (tests with partial overlap or mixed signals) by flagging uncertainty

Prompt gives confident retire/keep recommendation for borderline cases without noting ambiguity

Include 5 borderline cases in golden dataset; check that output includes uncertainty language or conditional recommendations

Zero invented test names, coverage paths, or stakeholder names not present in input

Output references test cases, code paths, or people not provided in the input context

Cross-reference all named entities in output against input; flag any additions not present in source data