Prompt Assembly Token Budget Regression Test Prompt

This prompt is for prompt engineering teams and AI operations engineers who need to catch token budget regressions across prompt versions before they reach production. The job-to-be-done is automated regression testing: given a new prompt version and a set of representative inputs, the system must compare token counts, budget compliance, and output quality against a known-good baseline. The ideal user is an engineer integrating this check into a CI/CD pipeline or a pre-release review process. Required context includes the baseline prompt, the candidate prompt, a test suite of inputs with expected output schemas, and defined budget thresholds per prompt section.

Use this prompt when your team ships prompt changes frequently, when context window costs are material, or when a prompt assembly pipeline combines multiple dynamic sections whose combined token count is unpredictable. It is also appropriate when you need objective, repeatable evidence that a prompt refactor did not silently inflate token usage or degrade output quality. The prompt produces a structured test report—not a one-off spot check—making it suitable for automated gating in a deployment pipeline.

Do not use this prompt for initial prompt design or ad-hoc debugging. It assumes you already have a baseline prompt and a candidate change. It is not a substitute for manual review of behavioral changes, safety policy adherence, or domain-specific correctness. If your test suite lacks edge cases or your budget thresholds are arbitrary, the regression test will produce a false sense of security. Pair this with behavioral eval prompts and human review for high-risk or regulated workflows. After reading this section, proceed to the prompt template to copy and adapt it for your own test harness.

This template is the core of the regression test workflow. It instructs the model to act as a test executor, comparing a new prompt version against a baseline. The model is given both versions, a set of test inputs, and strict evaluation criteria. The output is a structured test report, not a conversational response. The prompt is designed to be called programmatically in a CI/CD pipeline, so it prioritizes machine-readable output and unambiguous failure signals over narrative explanation.

textCopy
You are a prompt regression test executor. Your task is to compare a BASELINE prompt against a CANDIDATE prompt using a set of TEST_INPUTS. You will produce a structured test report in JSON format.

## BASELINE PROMPT (reference version)
[BASELINE_PROMPT]

## CANDIDATE PROMPT (new version under test)
[CANDIDATE_PROMPT]

## TEST INPUTS
[TEST_INPUTS]

## EVALUATION CRITERIA
For each test input, you must evaluate the following dimensions:
- **token_count**: The total token count of the assembled prompt (baseline vs. candidate). Use the [TOKEN_COUNT_FUNCTION] to calculate this.
- **budget_compliance**: Whether the token count is within the [TOKEN_BUDGET] limit. A value over the budget is a FAIL.
- **output_quality**: A pass/fail assessment based on the [QUALITY_RUBRIC]. The candidate output must be functionally equivalent to or better than the baseline.
- **regression_flag**: Set to TRUE if the candidate prompt uses more tokens than the baseline AND exceeds the budget, OR if the candidate output fails the quality rubric while the baseline passes.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure. Do not include any text outside the JSON object.
{
  "test_run_id": "[RUN_ID]",
  "baseline_version": "[BASELINE_VERSION]",
  "candidate_version": "[CANDIDATE_VERSION]",
  "overall_regression_detected": boolean,
  "results": [
    {
      "input_id": "string",
      "baseline_token_count": number,
      "candidate_token_count": number,
      "budget_compliance": "PASS" | "FAIL",
      "output_quality": "PASS" | "FAIL",
      "regression_flag": boolean,
      "notes": "string explaining any failure"
    }
  ]
}

## CONSTRAINTS
- You must evaluate every test input.
- If [TOKEN_COUNT_FUNCTION] is not available, estimate token counts using the [TOKEN_ESTIMATION_RULES].
- If the quality rubric cannot be applied confidently, flag the result for [HUMAN_REVIEW] and set output_quality to "FAIL".
- The overall_regression_detected field must be TRUE if any individual result has a regression_flag of TRUE.

To adapt this template, replace the square-bracket placeholders with concrete values from your test harness. [BASELINE_PROMPT] and [CANDIDATE_PROMPT] should be the fully assembled prompt strings, not just file paths. [TEST_INPUTS] should be a JSON array of input objects representing the dynamic variables that will be substituted into the prompts. The [QUALITY_RUBRIC] is critical for high-risk domains; it should define what functional equivalence means for your use case. If you are testing in a regulated environment, ensure the [HUMAN_REVIEW] flag is wired to a real review queue and not just logged. The [TOKEN_BUDGET] should be set to your production limit, and the [TOKEN_COUNT_FUNCTION] should be the exact tokenizer your application uses to avoid discrepancies between estimated and actual token counts.

This prompt is designed to be the evaluation step in a prompt versioning pipeline, not a one-off manual check. The harness should treat every prompt change as a candidate release and run this test suite before the change can merge. The prompt expects a [BASELINE_PROMPT], a [CANDIDATE_PROMPT], and a [GOLDEN_TEST_SET] of representative inputs. The output is a structured regression report that your CI system can parse to decide pass, warn, or fail. The core job of the harness is to call this prompt, parse its structured output, and enforce the thresholds defined in [BUDGET_THRESHOLDS] and [QUALITY_THRESHOLDS].

To integrate this into a CI/CD pipeline, wrap the prompt in a test runner script. The script should: (1) Load the baseline and candidate prompt templates from your version-controlled prompt registry. (2) For each input in the golden test set, render both prompts with identical variable substitutions and send them to the model with response_format set to json_object to enforce the output schema. (3) Parse the returned JSON and extract the token_count_delta, budget_compliance, and quality_regression_flag fields. (4) Fail the pipeline if budget_compliance is false for any test case or if the mean token_count_delta exceeds a hard limit (e.g., +15% tokens). (5) Emit a warning if quality_regression_flag is true for any case, requiring human review. Log the full report as a CI artifact for auditability. Use a model with a large context window and strong JSON mode support, such as gpt-4o or claude-3-5-sonnet, to ensure the prompt itself doesn't suffer from the budget problem it's testing.

The most common production failure is a false negative where a legitimate prompt improvement increases token count for good reason (e.g., adding necessary safety instructions). To prevent this, the harness must allow an override list in [APPROVED_BUDGET_INCREASES] where specific sections or test cases can have higher thresholds. A second failure mode is test set staleness: if the golden inputs no longer represent production traffic, the regression test becomes a meaningless gate. Schedule a recurring job to sample recent production inputs, cluster them, and update the golden set. Finally, never let this test become the sole quality gate. Pair it with behavioral eval runs and a human review step for any prompt that triggers a quality regression flag, especially in regulated domains where output accuracy has compliance implications.