Model Card Claim vs. Benchmark Result Contradiction Prompt

This prompt is designed for ML engineers and model auditors who need to verify that a model card's stated capabilities, limitations, and performance metrics are consistent with actual benchmark evaluation results. Use it when you have a model card document and a set of benchmark run outputs, and you need a structured discrepancy report that flags contradictions, unsupported claims, and methodology gaps. This is not a general fact-checking prompt; it assumes domain-specific knowledge of ML evaluation practices, including statistical significance, dataset contamination risks, and metric definitions. The prompt works best as part of a model release or procurement review pipeline where documentation accuracy is a governance requirement.

The ideal user is someone who understands evaluation methodology and can provide both the model card text and the raw benchmark outputs with sufficient metadata (dataset versions, metric implementations, significance thresholds, and contamination analysis results). The prompt expects structured inputs: a model card section with explicit claims about capabilities or performance, and corresponding benchmark results that include metric values, confidence intervals, and test conditions. Without both inputs, the prompt cannot perform meaningful contradiction detection. Do not use this prompt for general-purpose fact-checking, for evaluating claims without benchmark evidence, or for assessing qualitative claims that lack measurable performance dimensions.

Before running this prompt, ensure your benchmark results include the same metrics referenced in the model card, with clear documentation of evaluation methodology. Common failure modes include metric name mismatches (e.g., 'accuracy' vs. 'top-1 accuracy'), missing significance tests, and unstated dataset contamination risks. The prompt will flag these gaps, but you should pre-process inputs to normalize metric names and align evaluation conditions where possible. For high-stakes model release decisions, always pair this automated check with human expert review of flagged discrepancies, as the prompt may miss subtle methodological differences that invalidate direct comparisons.

This section provides a copy-ready prompt template that you can paste directly into your orchestration layer. The prompt is designed to take a model card's stated claims and a set of benchmark evaluation results, then produce a structured discrepancy report. It identifies contradictions between what the model card asserts and what the benchmark data actually shows, paying special attention to statistical significance, evaluation methodology mismatches, and unsupported capability claims. The template uses square-bracket placeholders that you must replace with your actual data before sending to the model.

textCopy
You are an ML evaluation auditor. Your task is to compare claims made in a model card against actual benchmark evaluation results and identify contradictions, unsupported claims, and methodological discrepancies.

## INPUTS

### Model Card Claims
[CLAIMS]

### Benchmark Results
[BENCHMARK_RESULTS]

### Evaluation Context (optional)
[EVALUATION_CONTEXT]

## INSTRUCTIONS

1. Extract every factual claim from the model card that relates to model capabilities, performance, limitations, or comparisons. Treat each claim as a discrete, verifiable assertion.

2. For each claim, search the benchmark results for supporting or contradicting evidence. Consider:
   - Direct metric comparisons (exact match, F1, accuracy, etc.)
   - Relative performance claims ("outperforms X," "state-of-the-art on Y")
   - Capability assertions ("can handle Z," "robust to W")
   - Limitation statements ("struggles with A," "not suitable for B")
   - Fairness or bias claims
   - Efficiency or latency claims

3. For each claim, classify the relationship to evidence as one of:
   - **SUPPORTED**: Benchmark evidence directly confirms the claim within stated tolerances.
   - **CONTRADICTED**: Benchmark evidence directly refutes the claim.
   - **PARTIALLY_SUPPORTED**: Evidence partially confirms but with notable caveats, weaker magnitude, or narrower scope than claimed.
   - **UNVERIFIABLE**: No relevant benchmark evidence exists to evaluate the claim.
   - **METHODOLOGY_MISMATCH**: Evidence exists but uses different evaluation methodology, metrics, or conditions that prevent direct comparison.

4. For CONTRADICTED and PARTIALLY_SUPPORTED claims, calculate or note:
   - The magnitude of discrepancy (absolute and relative difference)
   - Whether the difference is statistically significant (if significance data is available)
   - Whether the discrepancy direction favors or harms the model card's position

5. Flag any methodological concerns:
   - Benchmark uses different data splits, preprocessing, or metric definitions than implied by the claim
   - Claim references a benchmark version or configuration not present in the results
   - Results lack statistical significance measures where they would be expected
   - Sample sizes or evaluation conditions differ materially

6. Identify any claims that appear to cherry-pick favorable results while ignoring contradictory evidence from the same benchmark suite.

## OUTPUT FORMAT

Return a JSON object with this structure:

{
  "summary": {
    "total_claims_extracted": <int>,
    "supported_count": <int>,
    "contradicted_count": <int>,
    "partially_supported_count": <int>,
    "unverifiable_count": <int>,
    "methodology_mismatch_count": <int>,
    "overall_assessment": "<CONSISTENT|MINOR_DISCREPANCIES|SIGNIFICANT_DISCREPANCIES|MISLEADING>"
  },
  "claims": [
    {
      "claim_id": "<string>",
      "claim_text": "<exact text or close paraphrase from model card>",
      "claim_category": "<PERFORMANCE|CAPABILITY|LIMITATION|COMPARISON|FAIRNESS|EFFICIENCY|OTHER>",
      "evidence_relationship": "<SUPPORTED|CONTRADICTED|PARTIALLY_SUPPORTED|UNVERIFIABLE|METHODOLOGY_MISMATCH>",
      "evidence_summary": "<concise description of relevant benchmark evidence>",
      "discrepancy_details": {
        "claimed_value": "<string or null>",
        "observed_value": "<string or null>",
        "absolute_difference": "<number or null>",
        "relative_difference_pct": "<number or null>",
        "statistically_significant": "<true|false|null>",
        "discrepancy_direction": "<CLAIM_OVERSTATES|CLAIM_UNDERSTATES|NOT_APPLICABLE>"
      },
      "methodology_notes": "<string or null, describe any methodological mismatches>",
      "confidence": "<HIGH|MEDIUM|LOW>"
    }
  ],
  "methodology_concerns": [
    {
      "concern_type": "<string>",
      "description": "<string>",
      "affected_claim_ids": ["<string>"]
    }
  ],
  "cherry_picking_indicators": [
    {
      "description": "<string>",
      "affected_claim_ids": ["<string>"]
    }
  ]
}

## CONSTRAINTS

- Do not invent benchmark results. Only use evidence present in [BENCHMARK_RESULTS].
- If a claim cannot be verified from available evidence, mark it UNVERIFIABLE rather than guessing.
- Distinguish between "no evidence exists" and "evidence exists but contradicts."
- When statistical significance data is absent, note this in methodology_concerns rather than assuming significance.
- Flag when claims use vague language ("state-of-the-art," "highly robust") without operationalized metrics.
- If [EVALUATION_CONTEXT] provides relevant methodology details, use them to assess methodology mismatches.

To adapt this prompt for your pipeline, replace the square-bracket placeholders with your actual data. [CLAIMS] should contain the full text of the model card or the specific claim sections you want to verify. [BENCHMARK_RESULTS] should include the evaluation output, ideally with metric values, confidence intervals, and methodology notes. [EVALUATION_CONTEXT] is optional but valuable for providing details about how benchmarks were run, which versions were used, and any known differences from the model card's stated evaluation setup. For production use, consider pre-processing the model card to extract claims before sending to this prompt, and post-validating the JSON output against the expected schema before ingesting it into your verification pipeline.

This prompt is designed as a single step within a larger model evaluation audit workflow. It takes a model card's claims and a set of benchmark results as structured inputs and produces a discrepancy report. The harness must ensure that the inputs are well-formed before the prompt is called and that the output is validated before it is consumed by a downstream reporting system or a human reviewer. Because model card claims can be vague ('state-of-the-art performance') and benchmarks can be noisy, the implementation should treat this prompt as a structured comparison engine, not a final arbiter of truth.

The recommended implementation pattern is a validate → prompt → validate → escalate loop. First, pre-process the model card text to extract discrete, verifiable claims using a separate claim extraction prompt. Pair each claim with the relevant benchmark result from a structured data store. The harness should then call this contradiction prompt with a single claim-benchmark pair per invocation to keep the context focused and the output schema reliable. The model's JSON output must be validated against a strict schema that requires fields like claim_id, benchmark_id, contradiction_type (an enum of direct_contradiction, partial_mismatch, methodology_gap, compatible), confidence_score (0.0-1.0), and a detailed_explanation. A retry layer should catch schema validation errors, and after a fixed number of retries (e.g., 3), the item should be flagged for human review. Log every input pair, raw output, and validation result for auditability.

For model choice, a capable mid-tier model like claude-3.5-sonnet or gpt-4o is appropriate for this task, as it requires nuanced reading comprehension and structured output discipline. Avoid using smaller, faster models that may struggle with the precise distinction between a genuine contradiction and a methodological difference in evaluation. The harness should also implement a statistical significance check: if a benchmark result's confidence interval overlaps with the model card's claimed range, the system should automatically set the contradiction_type to compatible or flag it for a separate numerical check before calling the LLM. The final output of the harness should be an aggregated discrepancy report, but every individual finding must remain traceable back to the raw prompt input and output for governance and reproducibility.