Claim Extraction Failure Mode and Edge Case Test Prompt

This prompt is a diagnostic tool for verification QA engineers and AI pipeline builders who need to stress-test their claim extraction prompts against a wide range of failure modes. It is not a claim extraction prompt itself. Instead, it generates a structured catalog of edge cases, adversarial inputs, and reproduction cases that you can feed into your own extraction prompts to measure their robustness. The primary job-to-be-done is answering the question: 'What will break first when real users or adversarial content hit my extraction pipeline?' The most dangerous failure mode in production is the false negative—a valid claim that the extraction prompt silently drops without any error signal. This playbook helps you find those gaps before they reach users.

Use this prompt when you are preparing to deploy a new extraction prompt, when you are evaluating a vendor or open-source extraction model, or when you have observed unexplained recall drops in production. You need a set of extracted claims from your own pipeline to serve as the [EXTRACTION_OUTPUT] input—the prompt works by comparing what your system produced against what it should have produced. Do not use this prompt as a substitute for running your extraction prompt against real production traffic; it complements production observability by generating targeted adversarial cases that real traffic may not surface for weeks or months. The ideal user has access to both the extraction prompt under test and a representative sample of source documents spanning the content types their pipeline will encounter.

This prompt requires you to provide your extraction prompt's output format specification in [OUTPUT_SCHEMA] so the generated failure cases match your pipeline's expected structure. You should also supply domain-specific terminology in [DOMAIN_CONTEXT] if your extraction pipeline targets legal, financial, medical, or other specialized content—generic failure mode catalogs miss domain-specific edge cases like regulatory hedging language or technical compound assertions. The prompt works best when you iterate: run the generated failure cases against your extraction prompt, catalog the actual failures, and feed those real failures back into a second round of adversarial generation to probe deeper. Avoid using this prompt once and assuming coverage is complete; adversarial discovery is iterative.

After generating the failure mode catalog, prioritize the cases by estimated production frequency and severity. Route high-severity cases (claims that would cause harm if silently dropped) to immediate prompt repair or to a human review fallback in your pipeline. Route low-severity cases to your regression test suite as golden examples. The most valuable output from this prompt is not the full catalog but the subset of cases that actually break your extraction prompt—those become your minimum viable test set for any future prompt changes. Do not skip human review of the generated cases; the prompt can produce implausible edge cases that waste engineering time if not triaged against your actual content distribution.

This prompt is designed for verification QA engineers who need to systematically probe a claim extraction prompt or pipeline for weaknesses before production deployment. It takes a target extraction prompt and a set of adversarial or edge-case inputs, then produces a structured failure mode catalog. Each entry includes the input that triggered the failure, the expected behavior, the observed behavior, and a severity classification. Use this before shipping any claim extraction system to catch silent claim drops, boundary errors, and misinterpretation of complex language.

textCopy
You are a verification QA engineer testing a claim extraction system for production readiness.

Your task is to stress-test the following claim extraction prompt against a curated set of adversarial and edge-case inputs. For each input, compare the extraction output against the expected ground-truth claims.

# TARGET EXTRACTION PROMPT UNDER TEST
[EXTRACTION_PROMPT]

# TEST INPUTS AND EXPECTED CLAIMS
[TEST_CASES]

# FAILURE MODE TAXONOMY
Use the following failure categories when classifying findings:
- SILENT_DROP: A valid claim present in the input is missing from the output entirely.
- BOUNDARY_ERROR: A claim is extracted but its boundaries are wrong (truncated, merged with adjacent claim, split incorrectly).
- ENTITY_ATTACHMENT_ERROR: A claim is present but linked to the wrong entity, speaker, or source.
- HEDGE_STRIPPING: Hedging language, uncertainty markers, or qualification terms are removed, changing the claim's certainty.
- COREFERENCE_FAILURE: A pronoun or implicit reference is not resolved, producing an incomplete or ambiguous claim.
- CAUSAL_MISATTRIBUTION: A causal or conditional relationship is extracted with the wrong direction, connector, or dependency.
- NUMERICAL_DISTORTION: A quantitative claim is extracted with altered values, units, tolerances, or comparison operators.
- TEMPORAL_MISALIGNMENT: Dates, durations, or temporal relationships are normalized incorrectly or stripped.
- OPINION_AS_FACT: An opinion, prediction, or subjective statement is extracted and presented as a verifiable factual claim.
- OVER_FRAGMENTATION: A single coherent claim is split into fragments that lose meaning individually.
- FALSE_COMPOUND: Unrelated statements are incorrectly linked into a single compound claim.
- PARAPHRASE_DRIFT: A paraphrased claim alters the original meaning or attribution.

# OUTPUT FORMAT
For each failure discovered, output a JSON object with the following fields:
{
  "failure_id": "string, unique identifier",
  "failure_category": "string, from taxonomy above",
  "severity": "CRITICAL | HIGH | MEDIUM | LOW",
  "input_text": "string, the exact input that triggered the failure",
  "expected_claims": ["array of expected claim strings"],
  "observed_output": "string, the raw extraction output",
  "reproduction_steps": "string, how to reproduce this failure",
  "root_cause_hypothesis": "string, likely cause in the extraction prompt or pipeline",
  "recommended_fix": "string, suggested prompt or pipeline change"
}

# SEVERITY GUIDELINES
- CRITICAL: Silent claim drop on a high-stakes factual assertion; would cause downstream verification to miss a dangerous falsehood.
- HIGH: Systematic boundary or entity errors affecting multiple claim types; produces misleading verification results.
- MEDIUM: Occasional hedge-stripping or coreference failures on ambiguous inputs; degrades confidence but doesn't silently drop claims.
- LOW: Minor formatting inconsistencies or over-fragmentation that doesn't change claim truth value.

# INSTRUCTIONS
1. For each test case, run the target extraction prompt and capture the output.
2. Compare the output against the expected claims.
3. If a discrepancy exists, classify it using the failure taxonomy.
4. Assign a severity based on the impact to downstream verification accuracy.
5. Write clear reproduction steps so another engineer can replicate the failure.
6. Propose a root cause hypothesis tied to specific instructions in the extraction prompt.
7. Suggest a concrete fix (prompt rewording, additional examples, post-processing rule).
8. If the extraction prompt handles an edge case correctly, note it as a PASS with a brief explanation of why it succeeded.

# ADDITIONAL CONSTRAINTS
- Do not dismiss borderline failures as 'acceptable.' Flag them and let the team decide.
- If the extraction prompt produces claims not in the expected set, check whether they are valid but missed by the test author (false positive in the test) or hallucinations (true failure).
- For inputs with nested claims, verify that all sub-claims are extracted, not just the outermost assertion.
- Pay special attention to inputs where the claim is spread across multiple sentences or paragraphs.

[RISK_LEVEL]

Adaptation guidance: Replace [EXTRACTION_PROMPT] with the exact system and user prompt you use for claim extraction, including any few-shot examples. Populate [TEST_CASES] with a structured list of inputs and their expected ground-truth claims. Build your test cases from known failure patterns: sentences with multiple embedded claims, statements with hedging language ('may,' 'could,' 'reportedly'), claims that span paragraph boundaries, numerical assertions with units, and inputs where the speaker or source shifts mid-paragraph. Include at least one adversarial input designed to trick the extraction prompt into fabricating a claim from opinion text. Set [RISK_LEVEL] to HIGH if the extraction pipeline feeds automated verification without human review; set to MEDIUM if a human-in-the-loop step exists downstream. Run this test prompt against multiple model versions and track the failure catalog over time as a regression gate.

This prompt is not a one-off debugging tool; it is a regression test generator that should be integrated into your prompt CI/CD pipeline. The primary harness concern is that claim extraction prompts often fail silently—dropping valid claims without raising an error. This test prompt is designed to surface those false negatives by producing a structured failure mode catalog that can be converted into automated eval assertions. Wire it to run against every new prompt version before release, treating the output as a set of test cases that must pass before deployment.

Implementation workflow: (1) Run the prompt against your candidate extraction prompt, feeding it a diverse set of adversarial inputs including nested claims, hedged statements, and compound assertions. (2) Parse the JSON output into a test harness that extracts each reproduction_case and expected_behavior. (3) For each case, execute your extraction prompt and assert that the output contains the expected claims (for false_negative modes) or excludes hallucinated claims (for false_positive modes). Use strict string matching with fuzzy fallback—exact match first, then semantic similarity with a threshold of 0.85 if exact match fails. (4) Log every mismatch with the failure mode ID, input text, expected output, and actual output for human triage.

Model choice and cost controls: Use a frontier model (GPT-4o, Claude 3.5 Sonnet) for generating the failure mode catalog because adversarial creativity requires strong reasoning. However, the subsequent regression runs against your extraction prompt can use a smaller model if your extraction prompt is designed for it. Set temperature=0.3 for the test generation prompt—high enough to produce varied edge cases, low enough to maintain structured output. Implement a token budget cap of 4,000 output tokens per run to control costs. For batch runs across multiple prompt versions, cache the generated test cases and only regenerate when the test prompt itself changes.

Validation and retry logic: Validate the JSON output against a strict schema requiring failure_mode_id, category, description, reproduction_case.input_text, and reproduction_case.expected_missed_claims fields. If validation fails, retry once with the validation error message appended to the prompt as a correction hint. If the retry also fails, log the raw output and escalate to a human reviewer. Do not silently accept malformed test catalogs—a broken test generator produces false confidence. For high-risk verification pipelines (legal, medical, financial), require human review of the generated failure mode catalog before it is committed to the regression suite.

Observability and traceability: Tag every generated test case with the prompt version hash, model identifier, and generation timestamp. Store the full prompt input and output in your prompt observability platform (LangSmith, Arize, or custom logging). When a regression test fails, the trace should link back to the exact failure mode that caught it, enabling rapid debugging. Avoid the trap of treating this as a manual QA step—the value comes from automation. Schedule weekly adversarial regeneration runs to catch drift as models and extraction prompts evolve.