Cost Optimization Side-Effect Eval Failure Prompt

This prompt is for FinOps-aware engineering teams who have recently deployed a cost-reduction change—such as a model downgrade, prompt prefix truncation, context window reduction, or cache strategy update—and are now observing a correlated drop in automated evaluation scores. The job-to-be-done is not just to confirm that quality degraded, but to produce a structured, evidence-backed trade-off analysis that quantifies which specific cost optimization action caused which specific eval dimension to fail, and by what margin. The ideal user is an AI reliability engineer, platform engineer, or technical lead who has access to both cost attribution data (token counts, model tier, cache hit rates) and eval run results (rubric scores, dimension-level pass/fail data) from the same production traces. Required context includes: the before-and-after cost profile, the eval results for both periods, the specific cost optimization change that was applied, and the acceptable quality thresholds defined by the product team.

This prompt should not be used for initial cost optimization planning, for real-time production gating, or for diagnosing eval failures that have no plausible connection to a cost change. It is a post-hoc diagnostic tool, not a live circuit breaker. It also should not be used when the eval failures are clearly caused by unrelated factors such as a model provider regression, a retrieval pipeline outage, or a prompt injection attack. The prompt assumes the reader has already ruled out these obvious causes and is specifically investigating the side effects of deliberate cost-reduction actions. The output is a decision-support artifact, not an automated rollback command; human judgment is required to decide whether the cost savings justify the quality regression based on the product's risk tolerance.

Before using this prompt, gather the following artifacts: a summary of the cost optimization change (e.g., 'switched from gpt-4o to gpt-4o-mini for the summarization step'), the per-step token consumption and cost before and after the change, the eval dimension scores before and after the change, and the product team's stated minimum acceptable thresholds for each eval dimension. The prompt is designed to work with structured trace data and eval logs, not raw conversation transcripts. If you lack per-step cost attribution or dimension-level eval scores, this prompt will produce speculative results and should not be relied upon for production decisions. The next step after receiving the analysis is to review the recommended thresholds with the product owner and decide whether to accept the regression, adjust the optimization, or roll back the change.

This prompt is designed for FinOps-aware engineering teams who need to understand whether a cost-reduction change—such as switching to a smaller model, reducing context window size, removing retrieval steps, or disabling a tool—caused specific eval dimension regressions. It forces a structured comparison between the pre-change and post-change states, maps cost savings to quality impacts, and recommends whether the trade-off is acceptable or requires rollback. The output is a decision brief, not just a score comparison.

textCopy
You are a cost-quality trade-off analyst for AI production systems. Your task is to investigate whether a cost-optimization change caused eval failures and to produce a structured trade-off analysis.

## INPUTS
- Pre-change configuration: [PRE_CHANGE_CONFIG]
- Post-change configuration: [POST_CHANGE_CONFIG]
- Cost delta: [COST_DELTA] (absolute or percentage change, with unit)
- Eval results before change: [PRE_CHANGE_EVAL_RESULTS]
- Eval results after change: [POST_CHANGE_EVAL_RESULTS]
- Eval rubric definitions: [EVAL_RUBRIC_DEFINITIONS]
- Production trace samples (pre-change): [PRE_CHANGE_TRACES]
- Production trace samples (post-change): [POST_CHANGE_TRACES]
- Acceptable quality thresholds per eval dimension: [QUALITY_THRESHOLDS]
- Business context and risk tolerance: [BUSINESS_CONTEXT]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "change_summary": {
    "optimization_type": "string (e.g., model_swap, context_reduction, tool_removal, retrieval_step_removed)",
    "cost_savings": "string (quantified savings with unit)",
    "change_description": "string (one-paragraph summary of what changed)"
  },
  "eval_dimension_impacts": [
    {
      "dimension": "string (eval dimension name from rubric)",
      "pre_change_score": number,
      "post_change_score": number,
      "delta": number,
      "direction": "degraded | improved | unchanged",
      "severity": "critical | major | minor | none",
      "trace_evidence": [
        {
          "trace_id": "string",
          "failure_mechanism": "string (specific cause in the trace)",
          "confidence": "high | medium | low"
        }
      ],
      "attribution_confidence": "high | medium | low",
      "attribution_rationale": "string (why this change likely caused the score shift)"
    }
  ],
  "trade_off_assessment": {
    "acceptable_dimensions": ["string (dimensions where degradation is within threshold)"],
    "unacceptable_dimensions": ["string (dimensions where degradation exceeds threshold)"],
    "net_assessment": "acceptable | conditional | unacceptable",
    "conditions_for_acceptance": ["string (specific conditions that would make this acceptable)"]
  },
  "recommendation": {
    "action": "accept | rollback | adjust_thresholds | further_investigation",
    "rationale": "string (evidence-based reasoning)",
    "rollback_trigger_conditions": ["string (specific conditions that would trigger rollback)"],
    "monitoring_plan": ["string (what to monitor going forward)"]
  },
  "evidence_quality": {
    "sample_size_adequacy": "adequate | borderline | inadequate",
    "confounding_factors": ["string (other changes or events that could explain the regression)"],
    "recommended_follow_up": ["string (additional data or tests needed)"]
  }
}

## CONSTRAINTS
- Do not attribute eval score changes to the cost optimization without trace evidence linking the specific change to the failure mechanism.
- Flag any confounding factors (e.g., simultaneous model provider updates, traffic pattern shifts, data drift) that could explain the regression independently.
- If sample sizes are too small for statistical confidence, mark the assessment as "conditional" and specify the minimum sample size needed.
- For each degraded dimension, identify at least one trace event that demonstrates the failure mechanism. If no trace evidence exists, set attribution_confidence to "low."
- Distinguish between acceptable paraphrasing variance and genuine quality regressions.
- If the cost optimization involved removing a retrieval step or tool, check whether the missing capability is the direct cause of eval failures.
- Recommend rollback if any critical eval dimension drops below its acceptable threshold and the degradation is confidently attributed to the change.
- Consider partial rollback or hybrid approaches (e.g., keeping the cheaper model for low-risk traffic but routing high-stakes requests to the original configuration).

## ANALYSIS STEPS
1. Parse the pre-change and post-change configurations to identify exactly what changed.
2. For each eval dimension, compare pre- and post-change scores and classify the direction and severity of change.
3. For each degraded dimension, examine the post-change traces to find specific failure mechanisms (truncated context, missing tool output, lower-quality generation, etc.).
4. Cross-reference trace evidence against the eval rubric to confirm that the observed failure mechanism would cause the score drop.
5. Identify any confounding factors that could explain the regression independently of the cost change.
6. Compare each dimension's degradation against the acceptable quality thresholds.
7. Produce a net assessment and actionable recommendation.

## RISK LEVEL: [RISK_LEVEL]
- If RISK_LEVEL is "high": require human review before accepting any recommendation. Flag all uncertainty explicitly. Recommend conservative thresholds.
- If RISK_LEVEL is "medium": provide confidence levels for all attributions. Recommend monitoring for 2x the evaluation period before final acceptance.
- If RISK_LEVEL is "low": proceed with recommendation but still flag any confounding factors.

Adapt this template by replacing each square-bracket placeholder with data from your observability and eval systems. The [PRE_CHANGE_EVAL_RESULTS] and [POST_CHANGE_EVAL_RESULTS] should include per-dimension scores, not just aggregate numbers. The [PRE_CHANGE_TRACES] and [POST_CHANGE_TRACES] should be sampled from the same time windows as the eval results and should include full step-level detail—truncated traces will produce low-confidence attributions. If you lack trace data, set attribution_confidence expectations accordingly and flag the evidence gap in your implementation harness. The [QUALITY_THRESHOLDS] must be defined before running this prompt; without them, the trade-off assessment cannot determine acceptability. Store these thresholds in your eval configuration, not in the prompt itself, and inject them at runtime.

This prompt is designed to be triggered automatically when a cost-optimization change (such as a model downgrade, prompt compression, or context window reduction) is deployed and a subsequent eval run detects a statistically significant drop in quality scores. The harness should receive the pre-change and post-change eval distributions, the cost delta, and the specific optimization action as structured inputs. Do not run this prompt on every individual eval failure; it is meant for aggregate regression analysis after a known cost intervention.

The implementation should follow a trigger → gather → analyze → decide loop. First, a CI/CD pipeline or feature flag system detects a cost-change deployment and initiates a canary eval run. If the eval gate fails, the harness gathers: (1) the eval results from both the baseline and canary periods, broken down by dimension; (2) the cost-per-request or total token consumption delta; (3) the specific optimization action taken (e.g., 'switched from gpt-4o to gpt-4o-mini', 'truncated context to 4K tokens', 'removed 3 few-shot examples'). These are injected into the prompt's [BASELINE_EVAL_RESULTS], [POST_CHANGE_EVAL_RESULTS], [COST_DELTA], and [OPTIMIZATION_ACTION] placeholders. The model's output—a structured trade-off matrix and recommendation—should be parsed into a JSON schema and logged to your observability platform alongside the deployment record.

Validation and guardrails are critical because this analysis can be used to justify shipping degraded quality. The harness must enforce: (a) the eval results input must contain dimension-level scores, not just a single aggregate number; (b) the cost delta must be expressed in the same unit (e.g., USD per 1K requests) for both periods; (c) if the model recommends accepting the trade-off, the output must be routed to a human approval queue with a minimum 24-hour review window before the cost change is permanently merged. Implement a retry with temperature variation if the output fails to parse into the expected TradeOffMatrix schema, but escalate to human review after two failed attempts. Log the full prompt, response, and parsed recommendation to your trace store for auditability.

Model choice matters. This analysis requires strong reasoning and structured output discipline. Use a capable model (e.g., Claude 3.5 Sonnet, GPT-4o) with JSON mode or structured output enforcement. Avoid running this on a model that is itself a cost-optimized downgrade, as the analysis quality will compound the problem you are trying to diagnose. If your observability stack supports it, attach the raw trace IDs for the sampled eval failures so a human reviewer can drill into individual cases that the aggregate analysis flags as concerning. The next step after receiving the recommendation is to either approve the cost change with the accepted quality threshold, revert the change, or iterate on a more targeted optimization that preserves the degraded eval dimensions.