Synthetic Adversarial Claim Generation for Red-Teaming Prompt

This prompt is built for verification QA engineers and red-team testers who need to stress-test a fact-checking or claim verification pipeline before it reaches production. The job-to-be-done is dataset generation: you provide a domain, deception pattern, or edge-case category, and the prompt produces a labeled set of synthetic adversarial claims. Each claim includes the attack pattern used, the expected verification behavior from a well-built pipeline, and failure-mode tags that tell you what to measure. Use this when you are onboarding a new domain, hardening an existing verification system, or suspecting coverage gaps in your current test suite. The output is a structured artifact for your eval harness, not a piece of content for end users.

To use this effectively, you must supply a clear domain context and specify which adversarial categories you want to probe—such as hedging language, false dichotomies, cherry-picked data, or temporal inconsistencies. The prompt works best when you constrain the output schema to match your test runner's expected format, including fields for claim_text, attack_pattern, expected_verification_behavior, and failure_mode_tags. You should also set a risk level that controls how subtle or aggressive the generated claims should be. For high-risk domains like healthcare or finance, always route the generated dataset through human review before it touches any model training or evaluation pipeline. The prompt is not designed to generate live content for end users, to train models on adversarial patterns without oversight, or to replace manual red-teaming by domain experts.

After generating the dataset, the next step is to feed these claims into your verification pipeline and measure the results against the expected behaviors defined in each record. Pay special attention to failure-mode tags: they tell you whether the system should have flagged a claim as unverifiable, detected a contradiction, or escalated for human review. If your pipeline silently passes claims tagged with false_negative_risk or overconfidence_risk, you have a coverage gap. Avoid using this prompt as a one-time check; treat it as a living test generator that evolves as you discover new failure modes in production.

This prompt template generates synthetic adversarial claims designed to stress-test verification pipelines. It produces claims across multiple deception patterns, ambiguity types, and domain-specific edge cases, each labeled with expected verification behaviors and failure-mode tags. The output is structured for direct ingestion into red-team test suites, allowing QA engineers to probe extraction boundaries, evidence-matching logic, and confidence-scoring calibration before deployment.

textCopy
You are an adversarial test-case generator for fact-checking and claim verification pipelines. Your task is to produce a diverse set of synthetic claims that stress-test a verification system's ability to handle edge cases, deception patterns, and ambiguous language.

## INPUT
[INPUT_DESCRIPTION]: A description of the verification pipeline under test, including its domain, evidence sources, and expected behavior.

## GENERATION PARAMETERS
- Claim count: [CLAIM_COUNT]
- Deception patterns to include: [DECEPTION_PATTERNS]
- Ambiguity types to include: [AMBIGUITY_TYPES]
- Domain focus: [DOMAIN]
- Difficulty distribution: [DIFFICULTY_DISTRIBUTION]

## OUTPUT SCHEMA
Return a JSON array of claim objects. Each object must follow this schema:
{
  "claim_id": "string, unique identifier",
  "claim_text": "string, the adversarial claim",
  "deception_pattern": "string, one of: [DECEPTION_PATTERNS]",
  "ambiguity_type": "string, one of: [AMBIGUITY_TYPES]",
  "domain": "string",
  "difficulty": "string, one of: easy, moderate, hard, expert",
  "expected_verification_behavior": "string, how a well-calibrated system should handle this claim",
  "failure_mode_tags": ["string, array of failure modes this claim is designed to trigger"],
  "ground_truth_label": "string, one of: true, false, unverifiable, partially_true, misleading",
  "required_evidence_fields": ["string, array of evidence types needed for proper verification"],
  "source_context_note": "string, any source context that would change the verification outcome"
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Generate claims that are realistic, varied, and designed to expose specific failure modes in the verification pipeline described.

Adapt this template by replacing each square-bracket placeholder with your target configuration. For [DECEPTION_PATTERNS], specify the deception categories you want to probe—such as paltering, false implication, cherry-picked data, or fabricated citations. For [AMBIGUITY_TYPES], define the ambiguity dimensions—referential ambiguity, scope vagueness, hedging language, or pronoun confusion. The [CONSTRAINTS] placeholder should include any domain-specific rules, such as avoiding real named entities, restricting claim length, or requiring source-mimicking formatting. The [EXAMPLES] placeholder should contain 2-3 few-shot examples that demonstrate the desired output shape and difficulty calibration. Set [RISK_LEVEL] to guide the adversarial intensity: low for routine edge-case testing, high for red-team exercises probing safety boundaries. After generation, validate every output against the schema before feeding claims into your test harness. Claims that fail schema validation should trigger a retry with the error message appended to the prompt.

The adversarial claim generation prompt is not a one-off playground exercise; it is a data factory for your verification test suite. To use it in production QA, you must wrap it in a harness that controls input variation, validates output structure, logs every generation run, and routes edge cases for human review. The harness should treat each prompt invocation as a single batch job: feed in a deception pattern, ambiguity type, domain, and risk level; receive back a structured claim set with expected verification behaviors and failure-mode tags; validate that the output conforms to the expected schema; and store the result in your red-team dataset with full provenance. Without this harness, you will accumulate unlabeled, unvalidated claims that are impossible to trace back to the test conditions that produced them.

Start by defining a strict output schema that the prompt must satisfy. For each generated claim, require fields such as claim_text, deception_pattern, ambiguity_type, domain, expected_verification_behavior (e.g., 'should flag as unverifiable', 'should detect contradiction', 'should request missing evidence'), failure_mode_tag (e.g., 'false_negative_risk', 'overconfidence_risk', 'context_stripping'), and difficulty_level. In your application code, parse the model response and validate every claim object against this schema before accepting it into your dataset. If validation fails, implement a retry loop with a maximum of two additional attempts, each time feeding the validation error back into the prompt as a correction instruction. If the third attempt still fails, log the raw output and route it to a human review queue rather than silently dropping it. This prevents silent data loss while keeping your dataset clean.

Model choice matters for this workflow. Adversarial claim generation requires creativity, domain knowledge, and an understanding of subtle deception patterns. Use a capable frontier model (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 2.0 Pro) for generation. Do not use smaller or quantized models for this task; they tend to produce repetitive, obvious claims that fail to stress-test your verification pipeline. For cost control, batch your generation runs by deception pattern and domain, and cache results keyed by the exact input parameters. If you regenerate claims with the same inputs, you should get different outputs due to sampling, but the cached set prevents accidental duplication in your test suite. Set temperature between 0.8 and 1.0 to encourage diversity; lower temperatures produce formulaic claims that are easy to verify and defeat the purpose of red-teaming.

Logging and traceability are non-negotiable. Every generated claim set must be stored with the full prompt input, model version, timestamp, and raw response. This lets you reproduce any generation run, audit for drift if a model update changes claim patterns, and attribute test failures to specific generation conditions. Store claims in a versioned dataset (e.g., a JSON Lines file in a git repository, or a database table with a generation_run_id column). Tag each claim with the prompt version that generated it. When you update the generation prompt, run a differential analysis: generate claims with the old and new prompt versions using the same input parameters, and compare the diversity, difficulty, and pattern coverage of the outputs. A prompt change that reduces claim diversity or eliminates entire deception patterns is a regression, not an improvement.

Finally, integrate human review at two points. First, periodically sample generated claims and have a domain expert rate them for adversarial realism and pattern accuracy. Use this feedback to tune the prompt's examples and constraints. Second, route any claim that the verification system under test handles incorrectly—especially false negatives where a deceptive claim passes as verified—back into the generation harness as a new example of what to produce more of. This closes the loop between red-teaming and prompt improvement. Avoid the temptation to fully automate this pipeline without review; adversarial claim quality drifts over time, and only human judgment can detect when the generator starts producing claims that are too easy, too formulaic, or no longer representative of real-world deception patterns your verification system will face in production.