Difficulty-Stratified Few-Shot Example Set Prompt

Use this prompt when you are constructing a training or evaluation example set and need to ensure your model encounters a systematic progression of difficulty. The primary job-to-be-done is preventing your model from overfitting to simple, homogeneous patterns that cause brittle failures on complex production inputs. This is critical for prompt engineers and AI builders who are moving beyond a handful of hand-crafted examples and need a structured, labeled dataset that explicitly teaches the model how to handle edge cases, long contexts, ambiguous constraints, and multi-step reasoning. If your production traffic includes a wide variance in user sophistication, input length, or task complexity, a flat example set will train a model that performs well on average but fails silently on the hardest 10% of requests.

You should not use this prompt when your task has a narrow, well-defined input distribution with low variance in difficulty. If all inputs are structurally identical—such as classifying single-sentence intents into five fixed categories—a stratified set adds unnecessary token overhead and complexity. Similarly, avoid this approach when you lack a clear definition of what constitutes 'easy,' 'moderate,' and 'hard' for your specific domain. The prompt requires you to supply concrete difficulty criteria and progression logic; without these, the generated tiers will be arbitrary and may introduce noise rather than signal. For high-risk domains like healthcare or legal review, the output of this prompt is a starting point for human curation, not a final dataset. Every generated example must be reviewed for factual accuracy and safety before it enters a production system prompt or fine-tuning dataset.

Before using this prompt, gather your seed examples, define your difficulty dimensions (e.g., input length, constraint count, ambiguity level, reasoning steps), and decide how many examples you need per tier. After generation, validate the output by running the stratified set through your model and measuring whether performance degrades gracefully across tiers. A sharp drop-off at the 'hard' tier indicates the examples are too difficult or poorly taught; a flat performance curve across all tiers suggests insufficient challenge variation. Use the companion Example Diversity Scoring Rubric Prompt and Coverage Gap Detection Prompt to audit the generated set before deployment. If you observe ceiling effects where the model fails on most hard examples, return to your difficulty criteria and adjust the progression logic rather than simply adding more examples.

The following template is designed to be pasted directly into your prompt engineering workspace. It instructs the model to produce a sequenced set of examples across easy, moderate, and hard difficulty tiers, each with a difficulty label and a brief justification for its placement. Replace every square-bracket placeholder with your specific task details before use. The template assumes you have already defined the core task and the desired output format; its job is to enforce stratified coverage and progression logic.

textCopy
You are an expert example curator for training AI systems. Your task is to generate a difficulty-stratified set of few-shot examples for the following task:

[TASK_DESCRIPTION]

Each example must be a complete input-output pair demonstrating the desired behavior. Produce exactly [TOTAL_EXAMPLE_COUNT] examples, distributed across three difficulty tiers as follows:
- Easy ([EASY_COUNT] examples): Straightforward cases with clear, unambiguous inputs and standard expected outputs.
- Moderate ([MODERATE_COUNT] examples): Cases with some complexity, such as multiple constraints, longer inputs, or edge-adjacent scenarios.
- Hard ([HARD_COUNT] examples): Challenging cases involving ambiguity, conflicting constraints, rare patterns, or boundary conditions.

For each example, output a JSON object with the following schema:
{
  "difficulty": "easy|moderate|hard",
  "input": "[INPUT_FIELD_DESCRIPTION]",
  "expected_output": "[OUTPUT_FIELD_DESCRIPTION]",
  "difficulty_rationale": "One sentence explaining why this example belongs in this tier."
}

Arrange the examples in progression order: all easy examples first, then moderate, then hard. Within each tier, order examples from simplest to most complex.

Before finalizing, verify:
1. No two examples are near-duplicates.
2. The hard tier includes at least one example that tests a known failure mode: [KNOWN_FAILURE_MODE].
3. The easy tier does not contain any examples that could be confused with moderate difficulty.
4. The set as a whole covers these required input variations: [REQUIRED_VARIATIONS].

Output the complete example set as a JSON array.

To adapt this template, start by filling in [TASK_DESCRIPTION] with a precise statement of what the model should do. Define [TOTAL_EXAMPLE_COUNT] and the per-tier counts based on your token budget and coverage needs—a common starting ratio is 40% easy, 40% moderate, 20% hard. The [KNOWN_FAILURE_MODE] placeholder should reference a specific production failure you have observed, such as 'misclassifying negated statements as positive' or 'hallucinating fields when the input is empty.' For [REQUIRED_VARIATIONS], list input dimensions that must appear, such as 'inputs under 10 words, inputs over 200 words, inputs with nested lists, and inputs with missing optional fields.' After generating the set, run it through your existing eval harness to confirm the hard-tier examples actually produce failures in your current prompt before you rely on them for improvement.

This prompt is not a one-off generator; it is a component inside a larger example curation pipeline. The implementation harness should treat the prompt as a deterministic specification that receives a fixed set of inputs and produces a structured, machine-readable output. The primary integration points are: (1) a seed example or task description, (2) a difficulty taxonomy (easy, moderate, hard) with definitions, (3) a target example count per tier, and (4) an output schema that downstream tooling can parse. Because the generated examples will be used to teach model behavior in production, the harness must enforce validation, versioning, and human review before any example set is deployed.

Wire the prompt into a pipeline function that accepts a configuration object: { seed_task, difficulty_definitions, examples_per_tier, output_format }. The function should call the LLM with the prompt template, parse the response into a typed structure (e.g., a list of Example objects with difficulty, input, expected_output, and rationale fields), and run a series of post-generation validators. Validation checks must include: (1) schema conformance—every example matches the expected fields and types; (2) difficulty distribution—the actual count per tier matches the requested count; (3) progression logic—hard examples genuinely require more steps, constraints, or reasoning than easy ones; (4) ceiling effect detection—if the hardest tier examples are still trivial, flag the set for human review; and (5) contradiction scanning—no two examples teach conflicting behaviors. Log every generation run with a unique example_set_id, the model version used, the raw prompt, the raw response, and all validation results. This audit trail is essential for debugging drift and regressions.

For model choice, use a model with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet). Set temperature low (0.0–0.2) to maximize reproducibility. If the model supports structured output mode (JSON mode or function calling with a strict schema), use it to enforce the output shape directly rather than relying on post-hoc parsing. Implement a retry loop with up to 3 attempts: if validation fails, feed the validation errors back into the prompt as [CORRECTION_NOTES] and request a repaired output. If retries are exhausted, escalate the failed generation to a human reviewer queue with the full context. For high-risk domains (healthcare, legal, finance), always require human approval before the example set enters any training or evaluation pipeline. Store approved example sets in a versioned registry with a status field (draft, validated, approved, deprecated).

Finally, integrate evaluation hooks that run before deployment. After the example set is generated and approved, use it in a separate evaluation prompt that measures whether the examples actually teach the intended behavior. Run a set of held-out test inputs through a model using the new examples and compare outputs against expected behavior. Track metrics: behavioral alignment score, failure rate on edge cases, and difficulty calibration accuracy. If the hardest examples don't improve model performance on hard test cases, the stratification is likely ineffective. Wire these metrics into your prompt observability stack so that example set quality is monitored alongside other prompt assets. Avoid the temptation to auto-approve generated examples; even with strong validation, example quality is a human judgment call that directly shapes model behavior in production.