Counterexample Injection Prompt for Ambiguous Inputs

This prompt is for classification and extraction system builders who are hardening their models against borderline, ambiguous, or adversarial inputs. The core job is to teach the model to recognize uncertainty and respond with calibrated caution—such as requesting clarification, outputting a low-confidence label, or abstaining—instead of guessing. The ideal user is an AI engineer or prompt architect who already has a working prompt for a structured task (e.g., entity extraction, intent classification, or policy routing) but is seeing confident, wrong answers on edge cases in production. You need a set of real or synthetic ambiguous inputs and a clear definition of the desired cautious behavior before using this playbook.

Do not use this prompt when your primary problem is missing schema fields, incorrect output formats, or basic instruction-following failures. Those issues belong to schema control and output repair workflows. This prompt is also not a substitute for a safety policy or refusal prompt—it teaches ambiguity handling, not harm refusal. If your system operates in a regulated domain (healthcare, legal, finance), you must pair this prompt with human review checkpoints and evidence-grounding requirements. The counterexamples you inject will define the boundary between acceptable uncertainty and unacceptable guesswork, so invest time in selecting examples that mirror your real production failure modes.

Before implementing, audit your recent production traces to identify the specific ambiguous input patterns that cause confident misclassifications. Gather 5–10 representative borderline cases. For each, define the correct cautious output: should the model ask a clarifying question, return a confidence score below a threshold, or output a special AMBIGUOUS label? These decisions become your counterexample set. After injecting this prompt section, run your full eval suite—including both clear-cut cases and the ambiguous cases you targeted—to verify that the model didn't become overly hesitant on straightforward inputs. The next section provides the copy-ready template you'll adapt with your specific counterexamples and output schema.

The following prompt template is designed to be copied directly into your prompt assembly pipeline. It forces the model to recognize ambiguity by presenting carefully chosen counterexamples—inputs that look similar to valid cases but should produce a different output, such as null, "unknown", or a structured uncertainty marker. The square-bracket placeholders must be replaced at runtime with your specific task configuration.

textCopy
You are a precise classification system. Your task is to analyze the input and produce a structured output according to the schema defined below.

## Task Description
[TASK_DESCRIPTION]

## Output Schema
You must respond with a valid JSON object matching this schema:
[OUTPUT_SCHEMA]

## Critical Rule: Handling Ambiguity
If the input is ambiguous, contradictory, or missing information required to make a confident determination, you must not guess. Instead, set the primary field to `null` and populate the `uncertainty_reason` field with a brief explanation of what is unclear. Do not default to the most common class.

## Counterexamples (What NOT to Do)
The following examples demonstrate ambiguous inputs and the correct way to handle them. Study these carefully before processing the user's input.

### Counterexample 1
**Input:** [COUNTEREXAMPLE_INPUT_1]
**Incorrect Output (DO NOT DO THIS):** [INCORRECT_OUTPUT_1]
**Correct Output:** [CORRECT_OUTPUT_1]
**Why:** [RATIONALE_1]

### Counterexample 2
**Input:** [COUNTEREXAMPLE_INPUT_2]
**Incorrect Output (DO NOT DO THIS):** [INCORRECT_OUTPUT_2]
**Correct Output:** [CORRECT_OUTPUT_2]
**Why:** [RATIONALE_2]

### Counterexample 3
**Input:** [COUNTEREXAMPLE_INPUT_3]
**Incorrect Output (DO NOT DO THIS):** [INCORRECT_OUTPUT_3]
**Correct Output:** [CORRECT_OUTPUT_3]
**Why:** [RATIONALE_3]

## Positive Examples (What TO Do)
The following examples demonstrate clear inputs and their correct outputs.

### Example 1
**Input:** [POSITIVE_EXAMPLE_INPUT_1]
**Correct Output:** [POSITIVE_EXAMPLE_OUTPUT_1]

### Example 2
**Input:** [POSITIVE_EXAMPLE_INPUT_2]
**Correct Output:** [POSITIVE_EXAMPLE_OUTPUT_2]

## User Input
[USER_INPUT]

## Your Response (JSON only, no markdown fences)

To adapt this template for your own use case, replace the [TASK_DESCRIPTION] and [OUTPUT_SCHEMA] placeholders with your specific classification or extraction instructions and the exact JSON schema you expect. The counterexample blocks are the core teaching mechanism: each one must pair an ambiguous input with both an incorrect guess and the correct abstention or uncertainty output. Select counterexamples that are maximally confusing—inputs that sit right on your decision boundary. If your task has multiple ambiguity modes (e.g., missing entities, conflicting signals, out-of-distribution inputs), include at least one counterexample per mode. The positive examples should reinforce the desired behavior for clear-cut cases. Before deploying, run this prompt against a golden eval set that includes known ambiguous inputs and verify that the model's uncertainty_reason field is populated correctly and that primary fields are null when expected.

The counterexample injection prompt is not a standalone artifact; it is a pre-processing step that runs before the main classification or extraction prompt. In a production harness, you should assemble the counterexample block dynamically based on the input's ambiguity score. When an upstream ambiguity detector or low-confidence classifier flags an input, the harness retrieves relevant counterexamples from a curated library, injects them into the prompt template, and re-runs inference. This two-pass architecture keeps the common case fast while adding targeted instruction for borderline inputs.

Wire the prompt into your application as a conditional branch. First, run the primary classifier with a confidence threshold. If confidence falls below the threshold (e.g., <0.85 for a softmax classifier or the model outputs an explicit low-confidence token), trigger the counterexample injection path. In this path, query your example library using the input text and the top-k predicted classes. Insert the retrieved counterexamples into the [COUNTEREXAMPLES] placeholder of the prompt template. Use a structured format for each counterexample: Input: [ambiguous text] → Incorrect: [wrong output] → Correct: [right output] → Why: [boundary explanation]. After injection, re-run inference and compare the new output against the original. Log both passes, the injected examples, and the confidence delta for observability.

Validation must check three things. First, verify that the model did not simply flip to another high-confidence wrong answer—compare the new output against a known ground-truth set if available. Second, confirm that the output includes an explicit ambiguity handling signal (e.g., a needs_review flag, a confidence score, or a structured abstention marker) rather than a guess dressed in confident language. Third, enforce a retry budget: if the model still produces low-confidence or inconsistent outputs after two counterexample injection attempts, escalate to a human review queue. For high-risk domains like healthcare or legal classification, always require human sign-off on any output that passed through the counterexample injection path, regardless of the final confidence score.

Model choice matters. Counterexample injection works best with instruction-following models that attend carefully to in-context examples (e.g., GPT-4, Claude 3.5, Gemini 1.5 Pro). Smaller or older models may ignore counterexamples or treat them as positive examples, worsening the ambiguity problem. Test your chosen model's counterexample sensitivity by running a calibration set: inject counterexamples for known ambiguous cases and measure whether the model shifts its output in the correct direction. If the model consistently ignores counterexamples, consider fine-tuning on a dataset that includes explicit boundary demonstrations instead of relying on in-context injection.

Log every counterexample injection event with the input text, ambiguity trigger reason, retrieved examples, pre-injection output, post-injection output, confidence scores, and final disposition (accepted, escalated, or auto-resolved). This log becomes your evaluation dataset for tuning the ambiguity threshold, curating the example library, and detecting when counterexamples become stale or counterproductive. Run a weekly audit query: find cases where counterexample injection made the output worse, and remove or rewrite those examples.