Automated Test Case Selection for CI Runs Prompt Template

This prompt is designed for ML engineers and platform teams who need to reduce CI pipeline duration without sacrificing regression detection. It selects a minimal, high-signal subset of test cases from a golden dataset by analyzing the prompt diff, historical failure patterns, and a defined risk profile. Use this when your full test suite takes too long to run on every commit and you need an automated, evidence-based way to choose which tests to execute. The ideal user has a curated golden dataset with labeled failure histories and a clear understanding of their risk tolerance for different types of prompt changes.

This is not a replacement for running the full suite on release candidates. It is a gating optimization for pre-merge and early-stage CI checks. Do not use this prompt when the prompt diff is trivial, such as a typo fix with no semantic impact, or when the golden dataset lacks historical failure data, as the selection logic depends on that signal. It is also inappropriate for safety-critical prompt changes where a missed regression could cause user harm; in those cases, run the full suite. The prompt works best when integrated into a CI workflow that can automatically execute the selected subset, collect results, and feed failure data back into the historical record for future selection accuracy.

Before wiring this into your pipeline, ensure your golden dataset includes per-case metadata: a unique ID, the prompt version it was last run against, a failure count, and a risk tag. The prompt expects these fields to weight its selection. Start by running it against a known prompt diff and manually reviewing the selected subset for face validity. If the selection misses a known high-risk area, adjust the risk profile or add explicit constraints. Once validated, automate the selection as a pre-merge gate, but always run the full suite on release candidates and log any regressions that the subset missed to improve future selections.

This template is designed to be pasted directly into your test orchestrator or evaluation harness. It instructs the model to act as a CI/CD test selection engine, consuming a prompt diff, a golden dataset manifest, and a risk profile to output a prioritized, minimal subset of test cases. The goal is to maximize regression detection while minimizing CI pipeline runtime. All placeholders are enclosed in square brackets and must be replaced by your application logic before the model call.

textCopy
You are an automated test case selection engine for a CI/CD pipeline that evaluates prompt changes. Your task is to select the most relevant test cases from a golden dataset to run against a proposed prompt modification. Your selection must minimize total execution time while maximizing the probability of detecting a regression.

## INPUTS
- **Prompt Diff:** [PROMPT_DIFF]
- **Golden Dataset Manifest:** [DATASET_MANIFEST]
- **Historical Failure Map:** [FAILURE_MAP]
- **Risk Profile:** [RISK_PROFILE]

## SELECTION RULES
1. Analyze the [PROMPT_DIFF] to identify which instruction blocks, constraints, or output schemas were added, removed, or modified.
2. Map these changes to the test cases in the [DATASET_MANIFEST] that exercise those specific behaviors.
3. Cross-reference with the [FAILURE_MAP] to prioritize test cases that have historically detected regressions for similar changes.
4. Apply the [RISK_PROFILE] to adjust the selection: a "high-risk" profile should include more edge cases and adversarial examples, while a "low-risk" profile should favor a minimal smoke-test subset.
5. If the diff is empty or unparseable, default to the "sanity" subset defined in the [DATASET_MANIFEST].

## OUTPUT FORMAT
Return a valid JSON object with the following schema:
{
  "selected_test_ids": ["string"],
  "selection_rationale": "string",
  "estimated_runtime_seconds": number,
  "coverage_gaps": ["string"]
}

## CONSTRAINTS
- Do not select more than [MAX_TEST_CASES] test cases.
- Do not select test cases tagged as "manual" or "destructive" in the manifest.
- If the [RISK_PROFILE] is "critical", you must include all test cases tagged as "security" or "safety".
- Provide a clear, one-line rationale for each selected test case in the `selection_rationale` field.

To adapt this template, replace the placeholders with data from your pipeline context. The [PROMPT_DIFF] should be a unified diff of the prompt text. The [DATASET_MANIFEST] should be a structured list of test case IDs, descriptions, tags, and estimated runtimes. The [FAILURE_MAP] is a historical record linking past prompt changes to the test cases that caught the regression. The [RISK_PROFILE] is a string like "low", "medium", "high", or "critical" set by your release process. Before deploying this prompt, validate that the model's JSON output strictly conforms to the schema; a parsing failure should trigger a retry or fallback to a default test suite. For high-risk deployments, always log the selection_rationale for human review to ensure no critical coverage gaps were introduced by the model's selection logic.

The automated test case selection prompt is designed to sit at the very beginning of your CI evaluation stage, acting as a dynamic filter between a full golden dataset and the test runner. Instead of executing every test case on every commit, the harness calls this prompt with the prompt diff, historical failure logs, and a risk profile. The model returns a prioritized subset of test case IDs. Your CI orchestrator then reads that subset and executes only those evaluations, dramatically reducing pipeline duration while preserving regression detection power. The harness must treat the model's output as a recommendation that can be overridden by hard rules—for example, you should always include a small set of safety-critical test cases regardless of the model's selection.

Wire the prompt into your CI platform (GitHub Actions, GitLab CI, Jenkins) as a gate step that runs before the eval job matrix. The step should: (1) compute the prompt diff between the current commit and the last released version using git diff or your prompt registry API; (2) fetch the last N CI run results and failure counts per test case from your test database or artifact store; (3) assemble the risk profile from a config file that tags test cases by criticality, flakiness score, and domain; (4) call the LLM with these inputs and a strict output schema requiring a JSON array of test case IDs plus a confidence score per selection. Validate the response immediately: reject outputs that reference non-existent test case IDs, exceed a maximum selection count, or omit mandatory safety cases. If validation fails, retry once with the validation errors injected into the [CONSTRAINTS] block. If the retry also fails, fall back to a static subset defined in your risk profile config—never let a malformed LLM response block the pipeline.

Model choice matters here. This is a classification and ranking task that benefits from models with strong instruction-following and JSON mode support. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid smaller or older models that may hallucinate test case IDs or ignore the output schema. Set temperature=0 to maximize deterministic selection. Log every prompt call, the full response, validation results, and the final test subset to your observability platform. This trace is essential for debugging false negatives—when a regression slips through because the selector omitted the revealing test case. Over time, feed these misses back into the risk profile and historical failure data to improve selection accuracy. Do not use this prompt for compliance-mandated test suites where every case must run; the harness should have a bypass flag that forces full-suite execution for release branches or audit runs.