Contradictory Example Detection Prompt

This prompt is designed for teams managing large few-shot example libraries where silent contradictions degrade model behavior. It identifies example pairs that teach opposing behaviors for semantically similar inputs and produces a conflict report with severity scoring. Use this when you have more than 20 examples, when multiple authors contribute to the example set, or when you observe inconsistent model outputs that trace back to conflicting demonstrations. This is a QA workflow, not a runtime prompt. Run it during example set review, before prompt version releases, or as part of a CI pipeline for prompt changes.

The prompt works by performing pairwise comparison of examples, checking whether similar inputs produce contradictory outputs. It requires you to provide the full example set as [EXAMPLES], a similarity threshold [SIMILARITY_THRESHOLD] for determining when two inputs are semantically close enough to compare, and a contradiction severity scale [CONTRADICTION_SEVERITY_SCALE] that defines what counts as minor, moderate, or critical conflict. The output is a structured conflict report with conflicting pairs, severity scores, and recommended resolutions. You should configure the similarity threshold based on your domain's tolerance for variation—legal and medical domains typically need stricter thresholds than creative or conversational applications.

Do not use this prompt for small example sets under 20 items where manual review is faster and more accurate. Avoid it when examples are intentionally diverse and contradictory outputs are expected (e.g., teaching the model to handle ambiguous queries with multiple valid responses). If your example set has fewer than 10 examples, a manual pairwise review takes minutes and avoids the risk of automated false positives. For runtime contradiction detection in live model outputs, use a separate output consistency check prompt instead—this prompt is designed for pre-deployment QA, not production monitoring. After running this prompt, feed the conflict report into your example maintenance workflow and re-run after any example additions or modifications.

The following prompt template is designed to be copied directly into your prompt management system or codebase. It instructs the model to analyze a set of few-shot examples, identify pairs that teach conflicting behaviors for semantically similar inputs, and produce a structured conflict report. The template uses square-bracket placeholders for all dynamic inputs, making it easy to parameterize for different example sets, domains, and severity thresholds.

textCopy
Analyze the following few-shot example set and identify pairs that teach contradictory behaviors for semantically similar inputs. For each contradiction found, provide a severity score, the conflicting examples, and a recommended resolution.

Use the contradiction severity rubric below to assign scores. Only flag pairs where the input similarity exceeds the configured threshold.

[EXAMPLE_SET]

[CONTRADICTION_SEVERITY_RUBRIC]

[INPUT_SIMILARITY_THRESHOLD]

[DOMAIN_TAXONOMY]

Return your findings in the following JSON format:

[OUTPUT_SCHEMA]

To adapt this template, replace each placeholder with concrete values. [EXAMPLE_SET] should contain the full list of few-shot examples you want to audit, formatted consistently as input-output pairs. [CONTRADICTION_SEVERITY_RUBRIC] defines your scoring criteria—for example, 'Critical: outputs are logically incompatible; High: outputs recommend opposite actions; Medium: outputs differ in tone or constraint application; Low: minor stylistic divergence.' [INPUT_SIMILARITY_THRESHOLD] specifies the minimum semantic similarity score (e.g., 0.85 on a 0-1 scale) required to consider two inputs comparable. [DOMAIN_TAXONOMY] provides category labels that help the model understand your input space. [OUTPUT_SCHEMA] should be a valid JSON Schema or a clear description of the expected output structure, including fields like contradiction_id, example_pair, similarity_score, severity, conflict_description, and recommended_resolution.

Before deploying this prompt in production, validate that the output schema is strictly enforced—consider using structured output modes or a post-processing validation step that rejects malformed JSON. For high-stakes example sets where contradictions could cause safety incidents or compliance violations, route all detected conflicts to a human reviewer before accepting automated resolutions. Log every contradiction report with the prompt version, example set version, and timestamp for auditability. When integrating into a CI/CD pipeline for prompt releases, treat contradiction detection as a quality gate: block deployment if critical-severity conflicts are found and unresolved.

This prompt is designed for a batch processing pipeline, not a synchronous user-facing endpoint. For example sets exceeding 100 examples, implement a pre-processing step that groups examples by a domain taxonomy before sending them to the model. This prevents context window overflow and reduces token costs by ensuring the model only compares examples within the same behavioral category. The prompt's output is a structured conflict report, making it suitable for integration into a data quality workflow where the results are consumed by downstream systems rather than displayed directly to an end user.

To optimize cost and accuracy, run pairwise similarity computation outside the model using a text similarity library or embedding-based approach. Pre-filter candidate pairs so that only high-similarity examples are sent to the model for the more expensive contradiction judgment. This hybrid architecture keeps the model focused on nuanced behavioral analysis rather than brute-force similarity computation. Once the model returns a report, log all contradiction findings with their associated example IDs, severity scores, and a timestamp to create a durable audit trail. Integrate this check into your prompt version control system to automatically block merges when a CRITICAL contradiction is introduced. For operational workflows, set up a human review queue for HIGH severity contradictions and configure an auto-resolution policy for LOW severity ones where the semantic difference falls below a defined tolerance.

For CI integration, execute this prompt on every pull request that modifies example files. The pipeline should fail the check if any CRITICAL contradiction is detected, preventing the introduction of conflicting training signals. Ensure your harness includes robust error handling for model API failures and implements a retry strategy with exponential backoff. Before deploying, validate the end-to-end flow with a small, known set of contradictory examples to confirm that the pre-filtering threshold, model prompt, and post-processing logic produce the expected severity classifications.