Provider Upgrade Regression Smoke Test Prompt Template

This prompt is designed for platform and AI infrastructure teams who need a rapid, automated validation step immediately after a model provider upgrade. It acts as a smoke test, executing a minimal but high-signal set of capability probes across reasoning, tool calling, schema adherence, and refusal boundaries. The goal is not exhaustive regression testing but a quick pass/fail gate to catch catastrophic regressions before they reach downstream workflows or users. Use this when you have just pointed your application at a new model version or provider endpoint and need a structured, machine-readable health check before running a full test suite.

The ideal user is an engineering lead or MLOps engineer managing a model gateway or routing layer. Required context includes the new model endpoint, a reference set of capability probes, and the expected output schema for the pass/fail matrix. This prompt is not a replacement for a full regression suite, golden dataset evaluation, or production canary analysis. Do not use it as your only validation step for safety-critical applications, or when the upgrade involves a change in the fundamental model architecture that requires re-evaluating your entire prompt library. It is a first-line filter, not a comprehensive certification.

Before running this prompt, ensure you have a stable connection to the new provider endpoint and that your application's authentication and configuration are correct. The output is a structured JSON matrix designed to be parsed by an automated CI/CD gate. If any capability probe fails, the gate should block promotion and trigger a notification. For high-risk domains such as healthcare, finance, or safety systems, a human must review the failure diagnostics before any decision to roll back or proceed. After a passing smoke test, proceed to your full cross-model regression suite and production shadowing before declaring the upgrade safe.

This prompt template is designed to be the single smoke test you run immediately after upgrading a model provider or switching to a new model version. It executes a compact set of probes that stress the most common failure surfaces: reasoning, tool calling, schema adherence, and refusal boundaries. The output is a structured matrix you can parse programmatically to gate a release or trigger deeper investigation. Replace every square-bracket placeholder with your specific configuration before running.

textCopy
You are a regression smoke test executor. Your task is to run a minimal set of capability probes against the current model and return a structured pass/fail matrix.

## TEST CONFIGURATION
- Provider: [PROVIDER_NAME]
- Model Version: [MODEL_VERSION]
- Previous Baseline Version: [BASELINE_VERSION]
- Risk Level: [RISK_LEVEL]

## PROBE DEFINITIONS
Execute each probe exactly as specified. Do not skip probes. Do not add commentary outside the output schema.

### Probe 1: Reasoning
[REASONING_PROBE_INPUT]

### Probe 2: Tool Calling
Available tools:
[TOOLS]

Input: [TOOL_CALLING_PROBE_INPUT]

### Probe 3: Schema Adherence
Output schema:
[OUTPUT_SCHEMA]

Input: [SCHEMA_PROBE_INPUT]

### Probe 4: Refusal Boundary
Input: [REFUSAL_PROBE_INPUT]

## OUTPUT FORMAT
Return ONLY a valid JSON object matching this schema:
{
  "smoke_test_result": {
    "provider": "string",
    "model_version": "string",
    "baseline_version": "string",
    "execution_timestamp": "string (ISO 8601)",
    "overall_pass": "boolean",
    "probes": [
      {
        "probe_id": "string",
        "probe_name": "string",
        "status": "pass | fail | indeterminate",
        "output_summary": "string (max 200 chars)",
        "failure_mode": "string | null (reasoning_error | tool_selection_error | tool_argument_error | schema_violation | over_refusal | under_refusal | format_error | timeout | other)",
        "diagnostic_note": "string | null (specific error detail for debugging)",
        "regression_from_baseline": "boolean | null"
      }
    ],
    "critical_failures": ["probe_id"],
    "recommendation": "proceed | hold | rollback"
  }
}

## CONSTRAINTS
[CONSTRAINTS]

Adapt this template by populating the probe inputs with tasks that are representative of your production workload. For reasoning probes, use a multi-step logic problem your system encounters regularly. For tool calling, include the exact function definitions your agents use. For schema adherence, supply the strictest output schema your pipelines consume. For refusal boundaries, use borderline requests that sit near your safety policy thresholds. After pasting the populated prompt into your harness, validate the output JSON against the schema before parsing the overall_pass and recommendation fields. If any probe returns indeterminate, treat it as a failure for gating purposes and escalate for human review. Store the full result alongside your deployment record for auditability.

This prompt is designed to be executed as a gated step in a CI/CD pipeline, not as a one-off manual test. The implementation harness should treat the prompt as a parameterized test suite where the model endpoint, capability probes, and pass/fail thresholds are injected at runtime. The harness is responsible for calling the model, collecting the structured output, validating the pass/fail matrix, and blocking the pipeline if any critical capability regresses. Because this is a smoke test, the harness must enforce a strict timeout and cost ceiling—if the test takes more than 60 seconds or consumes more than a token budget defined in [MAX_TOKENS], the run should be marked as a failure to prevent runaway costs during automated upgrades.

The core integration pattern is a scripted runner that loads the prompt template, substitutes [MODEL_ENDPOINT], [CAPABILITY_PROBES], [PASS_THRESHOLD], and [OUTPUT_SCHEMA] from a configuration file or environment variables, then executes the call. After receiving the response, the harness must validate the JSON structure against the expected schema before interpreting results. A strict JSON schema validator should check for required fields (capability, status, diagnostics, evidence) and reject malformed outputs. If validation fails, the harness should retry once with a repair prompt that includes the validation error message. If the retry also fails, the test run is marked as a failure with a SCHEMA_VALIDATION_ERROR diagnostic. For each capability probe where status is FAIL, the harness should extract the diagnostics field and attach it to the pipeline log for immediate triage. The pass/fail gate logic should compare the count of PASS results against [PASS_THRESHOLD] and fail the pipeline if the threshold is not met or if any probe marked as CRITICAL in the configuration fails.

Model choice matters for this harness. The smoke test should be run against the specific model version being upgraded to, not a generic endpoint. If the provider supports model version pinning (e.g., gpt-4-0613 vs gpt-4-0125-preview), the harness must include the version tag in the request. For teams using a model gateway or router, the harness should bypass routing logic and call the target model directly to isolate provider-specific behavior. Logging must capture the full request payload, response payload, latency, token counts, and validation results for auditability. Store these artifacts in the CI/CD run artifacts or a dedicated evaluation store for post-upgrade analysis. If the smoke test fails, the harness should prevent automatic promotion of the new model version and trigger a notification to the on-call AI infrastructure channel with the failure diagnostics. Do not proceed with canary deployments or traffic shifting until the smoke test passes against the target model version.