Golden Set Expansion Prompt for Regression Testing

This prompt is for prompt QA teams and ML engineers who maintain a small, trusted set of 'golden' examples and need to scale them into a comprehensive regression test suite. The job-to-be-done is systematic coverage expansion: taking a handful of high-quality input-output pairs and generating a diverse, structured test set with expected outputs, pass/fail criteria, and coverage analysis. The ideal user already has a validated golden set and a defined evaluation rubric, and they need to catch regressions before a prompt change reaches production.

Use this prompt when your golden set is too small to catch subtle failures, when you are about to release a prompt version change and need a broader safety net, or when you want to measure coverage gaps across difficulty levels, edge cases, and input variations. The prompt requires several inputs: a seed set of golden examples with expected outputs, a target output schema, a list of known failure modes to test against, and a coverage dimension specification (e.g., difficulty, entity type, input length, ambiguity). Without these, the expansion will be unfocused and the resulting tests will not align with your actual regression risks.

Do not use this prompt as a substitute for creating the initial golden set. It expands existing quality, it does not invent it. If your seed examples are ambiguous, contradictory, or unvalidated, the expansion will amplify those problems. Also avoid this prompt when you need real user data for evaluation—synthetic expansion is a complement to production sampling, not a replacement. After running the expansion, always review the generated test cases for correctness before adding them to your regression suite. For high-risk domains where incorrect outputs could cause harm, require human approval on the expanded set and run a coverage gap analysis to identify what the expansion missed.

This prompt template takes a small set of validated input-output pairs (your golden set) and systematically expands them into a full regression test suite. The model generates variations that cover edge cases, boundary conditions, entity substitutions, and difficulty scaling while preserving the original intent and expected behavior. Each generated test case includes the input, expected output, pass/fail criteria, and a coverage category tag. Use this when you have 5-20 manually verified examples and need to scale to 100+ test cases before a prompt release.

textCopy
You are a regression test designer for AI prompt systems. Your task is to expand a small golden set of validated input-output examples into a comprehensive regression test suite.

## GOLDEN SET
[GOLDEN_SET]

## TASK DESCRIPTION
[TASK_DESCRIPTION]

## OUTPUT SCHEMA
[OUTPUT_SCHEMA]

## CONSTRAINTS
[CONSTRAINTS]

## VARIATION STRATEGIES
Apply these strategies to each golden example to generate test variants:

1. **Entity Substitution**: Replace names, dates, values, and domain terms with plausible alternatives while preserving structure.
2. **Paraphrase Rewriting**: Reword the input to test intent preservation across lexical variation.
3. **Boundary Probing**: Generate inputs at the edges of valid ranges (empty, maximum length, borderline values).
4. **Difficulty Scaling**: Create easier and harder versions by adjusting complexity, ambiguity, or reasoning depth.
5. **Negative Cases**: Generate inputs that should trigger refusal, clarification requests, or specific error handling.
6. **Format Perturbation**: Introduce minor formatting variations (whitespace, casing, punctuation) that should not change output.

## COVERAGE REQUIREMENTS
Ensure the expanded suite covers:
- All intent categories present in the golden set
- At least [MIN_EDGE_CASES] edge cases per category
- At least [MIN_NEGATIVE_CASES] negative test cases
- Input length range from [MIN_LENGTH] to [MAX_LENGTH] characters
- [REQUIRED_COVERAGE_DIMENSIONS]

## OUTPUT FORMAT
For each test case, produce a JSON object with these fields:
{
  "test_id": "unique identifier string",
  "source_golden_id": "reference to the golden example this derives from",
  "variation_strategy": "which strategy was applied",
  "input": "the test input text",
  "expected_output": "the expected model output or behavior",
  "pass_criteria": "specific conditions that define a passing result",
  "coverage_category": "intent, edge_case, negative, format, difficulty",
  "risk_level": "low, medium, high",
  "notes": "any special considerations for evaluation"
}

## ADDITIONAL INSTRUCTIONS
- Preserve the exact output schema and constraints from the golden set.
- Flag any generated test case where the expected output is uncertain with risk_level: "high" and explain why in notes.
- Do not generate test cases that violate [CONSTRAINTS].
- If the golden set contains fewer than [MIN_SEED_EXAMPLES] examples, note coverage gaps in a summary section.
- Include a coverage summary at the end listing which categories and strategies are represented and which gaps remain.

## RISK LEVEL
[RISK_LEVEL]

Adaptation guidance: Replace [GOLDEN_SET] with your validated input-output pairs in a structured format. [TASK_DESCRIPTION] should describe what the system under test is supposed to do. [OUTPUT_SCHEMA] defines the expected response format. [CONSTRAINTS] captures rules like refusal policies, length limits, or required disclaimers. Set [RISK_LEVEL] to "high" for regulated domains where human review of generated test cases is mandatory before they enter the test suite. For production use, always run the generated test cases through a validation step to confirm that expected outputs are correct and that negative cases genuinely represent failure conditions rather than acceptable alternative responses.

This prompt is designed to be called programmatically as part of a CI/CD pipeline for prompt QA, not as a one-off chat interaction. The implementation harness should treat the prompt as a deterministic transformation: a small set of golden examples goes in, and a structured regression test suite comes out. The harness is responsible for validating that the output is parseable, complete, and free of contamination before it is committed to the test suite. Because this prompt generates evaluation artifacts that will gate future prompt changes, the harness must enforce strict schema compliance and flag any output that could silently degrade test quality.

Wire the prompt into an application by wrapping it in a function that accepts a list of golden examples and a configuration object specifying the desired expansion factor, coverage dimensions, and output schema. The function should: (1) assemble the prompt with the golden examples injected into the [GOLDEN_EXAMPLES] placeholder and the expansion parameters in [EXPANSION_CONFIG]; (2) call the model with a low temperature (0.0–0.2) to maximize determinism; (3) parse the response against the expected [OUTPUT_SCHEMA], rejecting any output that fails JSON schema validation; (4) run a set of structural checks—each generated test case must have a unique ID, a non-empty input, an expected output, and at least one pass/fail criterion; (5) detect contamination by checking that no generated test case is a verbatim copy of a golden example; and (6) log the expansion run with a versioned artifact that includes the prompt hash, model version, timestamp, and coverage statistics. If any validation step fails, the harness should retry once with a revised prompt that includes the specific validation error in the [CONSTRAINTS] field, then escalate to human review if the second attempt also fails.

For high-risk domains where regression test quality directly impacts safety or compliance, add a human review gate before the expanded test suite is merged into the canonical test set. The review interface should display each generated test case alongside its source golden example, the coverage dimension it addresses, and any automated validation warnings. Reviewers should confirm that expected outputs are correct, pass/fail criteria are unambiguous, and no hallucinated assertions have been introduced. After approval, store the expanded suite with provenance metadata linking each test case to its generation run. Avoid wiring this prompt directly into an automated release gate without human review until you have measured the false-pass and false-fail rates of the generated tests against a manually curated holdout set over at least five expansion runs.