Adversarial Synonym Substitution Prompt

This prompt is for QA engineers and AI reliability teams who need to verify that a classification or routing prompt is stable under lexical perturbation, not just under clean, expected inputs. The job-to-be-done is systematic brittleness detection: you have a target prompt that classifies, routes, or labels user inputs, and you need to know whether it relies too heavily on exact keyword matching rather than semantic understanding. The ideal user is someone who owns a prompt testing pipeline, maintains a golden dataset, or is responsible for a release gate that prevents fragile prompts from reaching production. Required context includes the target prompt's full system and user message templates, a set of representative baseline inputs with their expected classifications, and a defined consistency threshold below which a prompt is considered too brittle to ship.

Do not use this prompt when you are testing for safety policy adherence, instruction hierarchy violations, or multi-turn conversational drift—those require specialized adversarial probes designed for injection, extraction, or state manipulation. This prompt is also not a replacement for a full semantic similarity evaluation; it generates perturbed test cases, but you still need a separate evaluation harness to run the target prompt against those cases and measure output consistency. The prompt assumes you have a classification schema with discrete, well-defined labels. If your target prompt produces free-text, ranked lists, or generative outputs, the consistency scoring logic must be adapted to use a semantic judge rather than exact-match comparison. Finally, this prompt is a test-generation tool, not a real-time defense. It helps you find brittle instructions before deployment, but it does not harden the prompt against adversarial inputs at runtime.

Before using this prompt, ensure you have a baseline set of at least 20-30 representative inputs with ground-truth labels. The adversarial synonym substitution is only meaningful if you can compare the perturbed output against a known expected classification. After generating the test matrix, run every perturbed input through your target prompt and measure the classification consistency rate. Any label flip on a semantically equivalent input is a brittleness signal that the instruction wording needs hardening. The next step is to feed the identified failure cases into a prompt debugging workflow, not to add more few-shot examples that overfit to the specific synonyms that broke it.

This prompt template is designed to be dropped into a QA harness that tests whether a target classifier or routing prompt is brittle to exact phrasing. It instructs the model to take a set of seed inputs and produce semantically equivalent variants using synonym substitution, paraphrasing, and syntactic reordering. The output is a structured test matrix that maps each variant back to its original input and expected classification, enabling automated consistency scoring.

textCopy
You are a QA engineer testing the robustness of a classification system. Your task is to generate adversarial test inputs that are semantically equivalent to the provided seed inputs but lexically different. Use synonym substitution, paraphrasing, and syntactic reordering to create variants that should produce the same classification as the original.

## INPUTS
- Seed inputs: [SEED_INPUTS]
- Target classification labels: [CLASSIFICATION_LABELS]
- Number of variants per seed input: [VARIANTS_PER_SEED]
- Perturbation intensity (low, medium, high): [PERTURBATION_INTENSITY]

## CONSTRAINTS
- Each variant must preserve the original semantic meaning and intended classification.
- Do not change entities, numbers, dates, or proper nouns unless substituting with a known synonym.
- For high intensity, you may reorder clauses and use less common synonyms, but meaning must remain intact.
- Flag any variant where you are uncertain about semantic equivalence with `uncertainty: true`.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "test_matrix": [
    {
      "seed_id": "string",
      "original_input": "string",
      "expected_classification": "string",
      "variants": [
        {
          "variant_id": "string",
          "variant_text": "string",
          "perturbation_type": "synonym_substitution | paraphrase | reordering",
          "uncertainty": boolean
        }
      ]
    }
  ]
}

## EXAMPLES
Seed input: "The customer wants to cancel their subscription immediately."
Expected classification: "cancel_request"
Variants:
- "The user is requesting immediate termination of their subscription." (paraphrase)
- "The client wants to end their subscription right now." (synonym_substitution)
- "Immediately cancel the subscription, the customer requests." (reordering)

To adapt this template, replace the square-bracket placeholders with your specific test configuration. [SEED_INPUTS] should be a JSON array of strings drawn from your golden dataset or production logs. [CLASSIFICATION_LABELS] is the set of valid labels your classifier can output. [VARIANTS_PER_SEED] controls test density—start with 3-5 for initial runs and increase for high-assurance gates. [PERTURBATION_INTENSITY] should be set to low for smoke tests and high for red-team runs. After generation, pipe the test_matrix into your classifier and compare each variant's predicted label against expected_classification. Flag any mismatch as a potential brittleness finding. For high-risk classification workflows such as safety routing or compliance triage, add a human review step before accepting variants with uncertainty: true into your regression suite.

This prompt is designed to run as a batch test generator, not a real-time production component. The typical integration pattern is a CI/CD pipeline step that executes the prompt against a target classification prompt, collects the generated test matrix, and then feeds each perturbed input through the classifier under test to measure consistency. The harness should treat the prompt output as a structured test specification that drives downstream evaluation, not as a final pass/fail verdict.

Wiring steps: 1) Load the target classification prompt and its label schema into [TARGET_PROMPT] and [LABEL_SET]. 2) Provide a representative sample of production inputs in [SEED_INPUTS]—aim for 20-50 examples covering each expected class. 3) Set [PERTURBATION_INTENSITY] to control how aggressively synonyms and paraphrases are applied (low, medium, high). 4) Call the adversarial synonym prompt and parse the output as a JSON test matrix where each row contains original_input, perturbed_input, expected_label, and perturbation_type. 5) Run each perturbed_input through the target classifier and compare the predicted label to expected_label. 6) Calculate consistency score per input (1.0 if all perturbations retain the original label, lower otherwise) and aggregate by class to identify brittle categories. Flag any input where consistency drops below [CONSISTENCY_THRESHOLD] (default 0.8) for manual review.

Validation and safety: Before executing, validate that the generated test matrix contains no empty perturbations, no duplicate rows, and that every expected_label exists in [LABEL_SET]. Log the perturbation count per seed input—if any seed produces zero valid perturbations, the intensity or synonym coverage may be insufficient. For high-stakes classifiers (safety, compliance, medical routing), add a human review gate on any perturbation that changes the expected label before accepting it as a valid test case; the prompt can occasionally generate a synonym that genuinely alters meaning. Store the test matrix alongside the prompt version and model version in your regression database so you can track stability over time. Model choice: Use a capable instruction-following model (GPT-4o, Claude 3.5 Sonnet, or equivalent) for test generation—cheaper models often produce synonyms that unintentionally shift semantics, creating false positives in your consistency checks. The target classifier under test can be any model; this harness only cares about output label comparison.