Scaffolded Example Removal Prompt for Fading Support

This playbook is for teams that have validated a few-shot prompt in production and now need to reduce token costs and latency by removing examples without degrading output quality. The scaffolded removal approach treats examples as training wheels: you start with a full set, measure baseline performance, then progressively drop examples while monitoring quality at each stage. Use this when you have a stable few-shot prompt with 5 to 20 examples, a defined evaluation metric, and a need to ship a leaner zero-shot or minimal-shot version.

The prompt template below provides a complete example set with embedded removal instructions. It includes a removal schedule, performance threshold gates, and a harness for measuring degradation per removed example. Each example block is tagged with a removal priority, and the prompt instructs the model to maintain output quality as examples are stripped away. The harness tracks metrics like format compliance, accuracy, and hallucination rate at each removal stage, with automatic rollback if quality drops below your defined threshold.

Do not use this if your task requires dynamic example selection per input or if your examples encode critical domain knowledge that the base model lacks entirely. This approach assumes the model can internalize the pattern from fewer examples over time. If your examples contain irreplaceable domain rules, edge-case handling, or safety boundaries, consider example compression or instruction distillation instead. Before starting, ensure you have a stable evaluation pipeline with at least 50 test cases covering nominal, edge, and failure scenarios.

This template assembles the full scaffolded prompt with instructions, the current example set, and a removal-stage marker. The [EXAMPLE_BLOCK] variable is replaced programmatically at each stage of the removal schedule. Start with all examples, then progressively drop the lowest-salience examples as defined in your removal schedule. The prompt is designed so that the model's behavior is driven by the examples early in training, but the instructions alone must sustain performance once all examples are removed.

textCopy
SYSTEM: You are a precise classification assistant. Your task is to analyze a customer support ticket and output a structured JSON object with the fields: "intent" (string), "priority" (one of "low", "medium", "high", "critical"), and "summary" (string, max 150 characters).

Follow these rules:
- If the ticket mentions a complete outage or data loss, priority must be "critical".
- If the ticket mentions a workaround exists, priority cannot be "critical".
- If the intent is unclear, set intent to "unknown" and priority to "medium".
- The summary must be a single sentence that captures the core issue without quoting the user verbatim.

[EXAMPLE_BLOCK]

USER: Ticket: [INPUT_TICKET_TEXT]

ASSISTANT:

The [EXAMPLE_BLOCK] placeholder should be replaced with a formatted set of few-shot examples. Each example must follow the exact USER: and ASSISTANT: turn structure shown above. At stage 0, include all examples. At each subsequent stage, remove the example with the lowest salience score as determined by your evaluation harness. The prompt's instructions must be self-contained enough that the model can perform correctly even when [EXAMPLE_BLOCK] is empty. Before deploying any stage, validate that the output conforms to the expected JSON schema and that priority assignments match ground-truth labels on a held-out calibration set.

The scaffolded example removal prompt is not a one-shot generation request. It is a stateful workflow that must be integrated into your application's prompt assembly pipeline. The core idea is to treat the full example set as a training scaffold that your application progressively removes as the model demonstrates pattern internalization. This requires a harness that tracks performance metrics per example configuration, gates removal on threshold achievement, and can roll back to a richer example set if performance degrades. Without this harness, you risk removing examples too early and silently degrading production quality.

To implement this, maintain a configuration store that maps a scaffold_level integer to a specific subset of examples. Level 0 contains the full example set. Each increment removes one or more examples according to your removal schedule. Your prompt assembly function reads the current scaffold_level from a feature flag or database, selects the corresponding example subset, and injects it into the prompt template. After each generation, log the scaffold_level, the model response, and the evaluation scores to your observability platform. Run a periodic evaluation job that calculates a moving average of your key metric—such as format compliance, task accuracy, or citation fidelity—over the last N requests at the current level. If the moving average exceeds your promotion threshold for a sustained window, increment the level. If it drops below a degradation threshold, immediately roll back to the previous level and trigger an alert.

The removal schedule itself should be defined as a configuration file checked into version control alongside your prompt. A typical schedule removes the simplest, most redundant examples first, preserving edge-case and boundary-condition examples until the final stages. For instance, if you have ten examples covering nominal cases, near-boundary cases, and error-recovery patterns, your schedule might remove one nominal example at levels 1-4, then one near-boundary example at levels 5-7, leaving error-recovery examples until levels 8-9. Never remove all examples at once. Always leave at least one anchor example until you have sustained zero-shot performance across multiple evaluation windows. The harness should also support manual overrides: operators must be able to pin a scaffold_level, force a promotion, or trigger a rollback from a dashboard without deploying code.

Before deploying this harness to production, run a shadow evaluation on logged traffic. Replay recent requests through each scaffold_level and measure the performance curve. This gives you a baseline for setting promotion and degradation thresholds. Expect some variance: example removal often causes a small, temporary dip before the model stabilizes at the new level. Your thresholds should require sustained performance above the baseline, not instantaneous improvement. Finally, wire the harness into your CI/CD pipeline so that any change to the example set or removal schedule triggers a regression test across all scaffold_level configurations. This prevents a well-intentioned example update from breaking the removal trajectory.