Automated Example Refresh Workflow Prompt

This prompt is designed for ML platform teams building self-healing prompt pipelines where example staleness is a known failure mode. The core job-to-be-done is automated candidate generation: when a drift detection system flags a stale few-shot example, this prompt takes the original stale example, its documented teaching intent, and a sample of current production inputs, then outputs a replacement candidate that matches the current data distribution while preserving the original behavioral lesson. The ideal user is an MLOps engineer or prompt maintainer who has already instrumented drift detection and needs a reliable generation step before human review or automated validation gates.

The workflow requires three concrete inputs to function correctly. First, the stale example must be provided in its original format, including the input-output pair and any associated metadata such as the example's ID, category, or creation date. Second, the teaching intent must be a clear, one-to-two-sentence description of what behavior the original example was designed to teach—for instance, 'handle refund requests where the order number is missing' or 'demonstrate polite refusal for out-of-scope medical advice.' Third, the production input sample should contain 10-50 recent, anonymized inputs drawn from the same operational context as the stale example, providing a statistical snapshot of the current distribution. The output is a structured candidate example with a similarity score, a preservation check against the teaching intent, and a human-readable justification for the replacement.

Do not use this prompt when the teaching intent itself has become invalid—for example, if a product change means a particular error-handling pattern is no longer needed, the correct action is to retire the example entirely rather than refresh it. Similarly, avoid this prompt when the production distribution has shifted so far that the original example category is no longer relevant; generating a replacement for a category that no longer exists in production wastes tokens and risks introducing misleading demonstrations. In high-stakes domains such as healthcare or legal workflows, always route the generated candidate through a human reviewer or an automated validation step that checks the candidate against a golden evaluation set before it replaces the stale example in the prompt template. The next section provides the copy-ready prompt template you can adapt for your own drift-to-refresh pipeline.

This prompt template is the core of an automated example refresh pipeline. It takes a stale example and a sample of current production inputs, then generates a replacement example that preserves the original teaching intent while matching the vocabulary, structure, and difficulty of real user traffic. Use this when your drift detection system has flagged specific examples for replacement and you want candidate examples before human review or automated insertion.

textCopy
You are an example curator for a production AI system. Your job is to replace a stale few-shot example with a new example that matches current production data while preserving the original teaching intent.

## INPUTS

**Stale Example (to replace):**
[STALE_EXAMPLE]

**Original Teaching Intent:**
[TEACHING_INTENT]

**Production Input Sample (recent real inputs):**
[PRODUCTION_SAMPLE]

**Output Format Required:**
[OUTPUT_SCHEMA]

**Constraints:**
[CONSTRAINTS]

## TASK

1. Analyze the stale example to understand what pattern, behavior, or boundary it was teaching.
2. Analyze the production input sample to understand current vocabulary, topic distribution, input length, and difficulty.
3. Generate exactly [CANDIDATE_COUNT] replacement candidate examples that:
   - Teach the same pattern as the original example.
   - Use vocabulary, topics, and structure consistent with the production sample.
   - Match the specified output format exactly.
   - Respect all constraints.
4. For each candidate, provide a brief explanation of how it preserves the original teaching intent while updating to current distribution.

## OUTPUT

Return a JSON object with this structure:
{
  "candidates": [
    {
      "id": "candidate-1",
      "input": <example input matching production style>,
      "output": <example output matching required format>,
      "intent_preservation": "Explanation of how this candidate teaches the same pattern.",
      "distribution_alignment": "Explanation of how this candidate reflects current production data."
    }
  ],
  "replacement_recommendation": {
    "primary_candidate_id": "candidate-X",
    "confidence": "high|medium|low",
    "rationale": "Why this candidate is the best replacement."
  }
}

## VALIDATION RULES
- Every candidate output must parse as valid [OUTPUT_SCHEMA].
- No candidate should be identical to the stale example.
- Candidates must not introduce new teaching intents not present in the original.
- If the production sample shows a distribution shift that makes the original teaching intent irrelevant, flag this in the recommendation with confidence "low" and explain why.

Before inserting any generated example into your prompt, validate that the output parses correctly against your schema, that the teaching intent is preserved, and that the example doesn't introduce contradictions with other examples in your set. For high-risk domains, route candidates through human review before production deployment. Run a regression test against your golden evaluation set to confirm the new example doesn't degrade performance on existing cases.

The automated example refresh prompt is not a one-shot script; it is a component inside a larger pipeline that detects drift, generates candidates, validates them, and safely inserts replacements. The harness must treat each generated candidate as untrusted until it passes structural, semantic, and behavioral checks. A typical pipeline begins with a drift detection trigger—such as a staleness score crossing a threshold or a coverage gap alert—which then invokes this prompt with the flagged stale example and a sample of recent production inputs that represent the current distribution.

Wire the prompt into an application by first constructing the input payload: the stale example's original input-output pair, the teaching intent (a short description of what the example was meant to demonstrate), and a batch of 5–20 recent production inputs sampled from the same intent cluster. The prompt returns one or more candidate replacement examples. Before insertion, run each candidate through a structural validator that checks for required fields, output schema compliance, and token length within budget. Then apply a semantic similarity check—embed the candidate's output against the stale example's output and flag candidates that diverge too far from the original teaching intent. Finally, run a behavioral regression test by inserting the candidate into a copy of the prompt and evaluating against a small golden test set. If the candidate causes any regression, discard it and retry with adjusted constraints.

For high-stakes production systems, add a human review gate before any example is inserted into the live prompt. The review interface should show the stale example, the production sample that triggered the refresh, the generated candidate, and the validation results. Log every refresh event with a timestamp, trigger reason, candidate scores, reviewer decision, and the before/after example pair. This audit trail is essential for debugging accuracy drops and for governance reviews. Choose a model with strong instruction-following and structured output capabilities—GPT-4o or Claude 3.5 Sonnet are good defaults. Set temperature low (0.0–0.2) to reduce creative drift in the generated examples. Implement a retry budget of 3 attempts per stale example; if all candidates fail validation, escalate to a human with the failure reasons rather than silently inserting a bad example. Never allow the pipeline to automatically replace more than 20% of the example set in a single run—batch refreshes reduce the blast radius of a bad generation.