Cross-Model Output Length Drift Detection Prompt Template

This playbook is for product and infrastructure teams who need to detect when a model upgrade, provider switch, or model version change causes outputs to become significantly longer or shorter. Output length drift breaks UI layouts, overflows text containers, alters user experience, and changes per-request costs. Use this prompt template to generate a structured statistical report comparing output length distributions across two or more models for identical inputs. The ideal user is an engineering lead or platform engineer preparing a model migration, running a CI/CD regression gate, or setting up periodic multi-model monitoring. You should have a representative sample of production prompts ready—ideally 100+ examples covering your typical input diversity—and access to both the source and target model endpoints.

This prompt is not a general quality evaluation tool. It does not assess semantic correctness, tone, or factual accuracy. It measures verbosity, token counts, character counts, truncation risk, and cost-impact estimates. Do not use this prompt when you need to evaluate whether outputs are factually correct or stylistically appropriate. Do not use it with fewer than 30 sample inputs, as small samples produce unreliable distribution estimates. If your application has strict maximum token limits that trigger hard truncation, pair this analysis with a separate semantic completeness check—a short output that is cut off mid-sentence may pass a length check but fail your users. For high-stakes migrations where output length directly impacts revenue or regulatory compliance, always supplement automated drift detection with human review of the most extreme outliers.

Before running this prompt, confirm that your input set is representative of production traffic and that you have instrumented both models to capture token counts from the API response metadata rather than estimating from character counts. After receiving the report, focus first on the truncation risk metrics and cost-impact projections. If the target model shows a statistically significant shift in median output length, investigate whether your UI can accommodate the new distribution before proceeding with the migration. The next step is to run a semantic equivalence check on the same input set to ensure that length changes do not correlate with meaning loss.

This prompt template is designed to be run once per model pair you are comparing. It accepts structured data containing the outputs from two models responding to identical prompts and produces a statistical drift report. The report quantifies verbosity changes, identifies which prompt categories are most affected, and estimates the downstream impact on token costs and UI truncation risk. Use this template when you are migrating between model versions, evaluating a new provider, or monitoring production drift over time.

textCopy
You are an output length drift analyzer. Your task is to compare the output lengths of two models responding to identical prompts and produce a structured drift report.

## INPUT DATA
You will receive a JSON array of comparison records. Each record contains:
- prompt_id: unique identifier for the prompt
- prompt_category: the functional category of the prompt (e.g., summarization, extraction, generation, classification)
- prompt_text: the full prompt sent to both models
- model_a_output: the complete output from Model A
- model_b_output: the complete output from Model B
- model_a_name: identifier for Model A
- model_b_name: identifier for Model B
- max_allowed_tokens: the UI or application token limit for this prompt category, if any

[INPUT_DATA]

## ANALYSIS INSTRUCTIONS
For each comparison record, compute:
1. Token count for model_a_output and model_b_output using the tokenizer specified in [TOKENIZER]. If no tokenizer is specified, estimate tokens as word_count * 1.3.
2. Absolute difference in tokens (model_b - model_a).
3. Percentage change relative to model_a.
4. Whether either output exceeds max_allowed_tokens (truncation risk flag).
5. Estimated cost impact per 1000 requests using [COST_PER_TOKEN_A] and [COST_PER_TOKEN_B].

Then aggregate across all records:
- Mean, median, and 95th percentile token counts per model.
- Mean absolute and percentage drift.
- Drift broken down by prompt_category.
- Count and percentage of records with truncation risk per model.
- Total estimated cost delta per 1000 requests.
- Statistical significance: report whether the drift is significant using a paired test appropriate for the distribution shape. Note if the distribution is non-normal.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "summary": {
    "model_a_name": string,
    "model_b_name": string,
    "total_records_compared": integer,
    "model_a_mean_tokens": float,
    "model_b_mean_tokens": float,
    "mean_absolute_drift_tokens": float,
    "mean_percentage_drift": float,
    "drift_direction": "model_b_longer" | "model_b_shorter" | "no_significant_drift",
    "statistical_significance": "significant" | "not_significant",
    "significance_test_used": string,
    "truncation_risk_count_model_a": integer,
    "truncation_risk_count_model_b": integer,
    "estimated_cost_delta_per_1k_requests": float
  },
  "per_category_breakdown": [
    {
      "category": string,
      "record_count": integer,
      "model_a_mean_tokens": float,
      "model_b_mean_tokens": float,
      "mean_drift_tokens": float,
      "mean_percentage_drift": float,
      "truncation_risk_model_a": integer,
      "truncation_risk_model_b": integer
    }
  ],
  "per_record_details": [
    {
      "prompt_id": string,
      "category": string,
      "model_a_tokens": integer,
      "model_b_tokens": integer,
      "absolute_drift": integer,
      "percentage_drift": float,
      "truncation_risk_model_a": boolean,
      "truncation_risk_model_b": boolean,
      "cost_delta": float
    }
  ],
  "alerts": [
    {
      "severity": "critical" | "warning" | "info",
      "message": string,
      "affected_categories": [string],
      "recommendation": string
    }
  ]
}

## ALERT THRESHOLDS
Generate alerts when:
- CRITICAL: Any category shows >50% mean drift or truncation risk increases by >20%.
- WARNING: Any category shows >20% mean drift or overall mean drift exceeds 15%.
- INFO: Overall drift is statistically significant but below warning thresholds.

## CONSTRAINTS
[CONSTRAINTS]

If no constraints are provided, apply these defaults:
- Do not hallucinate token counts. Use the provided tokenizer or the estimation formula.
- If an output is empty or missing, record token count as 0 and flag it in alerts.
- Report all numbers to 2 decimal places.
- If fewer than 10 records are provided, note that statistical significance may be unreliable.

To adapt this template, replace [INPUT_DATA] with your JSON array of comparison records. Set [TOKENIZER] to the tokenizer name if you have one available (e.g., cl100k_base for GPT-4, claude-3 for Claude). Provide [COST_PER_TOKEN_A] and [COST_PER_TOKEN_B] as floats representing the cost per token for each model. Use [CONSTRAINTS] to add domain-specific rules, such as ignoring outputs below a minimum token threshold or applying category-specific weightings. If you are comparing more than two models, run this template pairwise and merge the results. Always validate the output JSON against the schema before ingesting it into your monitoring dashboard or CI pipeline.

This prompt is designed to run as a scheduled batch job or CI gate step, not as a one-off manual check. The implementation harness should iterate over a fixed set of representative prompts—your golden input set—and send each to every model under test. Collect the raw outputs, then pass each output through a token-counting function (e.g., tiktoken for OpenAI models, equivalent tokenizers for others) to produce a per-model, per-input length measurement. The prompt template itself returns a structured JSON report, but the harness must also store the raw lengths for independent verification and time-series tracking.

Build the harness with three layers: execution, validation, and alerting. In the execution layer, use a queue or workflow engine (e.g., Temporal, Airflow, or a simple script with async HTTP calls) to fan out requests to each model endpoint. Include retry logic with exponential backoff for transient failures, but cap retries at 3 attempts to avoid masking persistent model unavailability. In the validation layer, parse the JSON output from the prompt and cross-check the reported statistics against your own token counts—never trust the model's self-reported lengths without verification. If the discrepancy exceeds 5%, flag the result and log both values. In the alerting layer, compare the current run's drift metrics against a stored baseline (e.g., the previous week's 95th percentile output length per model). Trigger an alert if the Jensen-Shannon divergence exceeds 0.15, if the truncation risk rate jumps above 2%, or if any model's mean output length shifts by more than 20% between consecutive runs.

Store every run's results in a time-series database or structured log (e.g., a model_drift_runs table with columns for run_id, model_id, input_id, token_count, truncated, cost_estimate, and the full prompt JSON report). This enables historical trend analysis and makes it easy to correlate drift events with model version updates or provider changes. For cost-impact estimates, wire the token counts into your actual pricing data per model—do not rely on the prompt's cost estimate alone, as pricing changes frequently. Finally, gate this check in your CI/CD pipeline for any model upgrade or provider migration: run the harness against the candidate model, compare against the current production baseline, and block the release if drift thresholds are breached. Avoid running this on every commit to your main prompt library; instead, trigger it on model version changes, provider API version bumps, or scheduled weekly cadences to catch silent model behavior shifts.