Evidence Matching Across Modalities Prompt

This prompt is designed for verification pipelines where a single claim must be checked against evidence that exists in multiple formats. A claim like 'revenue grew 14% in Q3' might need to be verified against a narrative paragraph in an earnings call transcript, a row in a financial table, and a structured data point from an API. Standard evidence-matching prompts fail here because they treat all evidence as unstructured text, losing the structural semantics of tables and the precision of typed data. This prompt instructs the model to perform format-aware comparison: it parses table structures, normalizes numerical values across representations, and maps chart-derived assertions back to their source data. Use it when your evidence corpus contains a mix of plain text, markdown tables, CSV snippets, JSON blobs, or structured data records, and you need a unified verification output.

The ideal user is a verification engineer or RAG pipeline builder who already has a claim extraction step upstream and an evidence retrieval system that returns heterogeneous results. You must provide the claim, the evidence set with format annotations, and a verification schema. The prompt expects evidence to be tagged by format type (e.g., text/plain, text/markdown-table, application/json, text/csv) so the model can apply the correct parsing strategy. Without format tags, the model will default to unstructured text comparison and lose the cross-modal precision this prompt is designed to deliver. You should also supply a tolerance window for numerical comparisons and a unit normalization map if evidence sources use different units or scales.

Do not use this prompt when all evidence is plain text, when claims are purely qualitative with no numerical or structural component, or when you cannot provide format metadata for each evidence chunk. In those cases, a standard claim-to-evidence pairing prompt with inline citation will be simpler and more cost-effective. Also avoid this prompt for real-time streaming verification where latency constraints prevent the multi-step parsing and comparison logic it requires. For high-stakes domains like financial reporting or clinical data, always route outputs through human review and log the full evidence-to-claim mapping for audit. If your evidence set exceeds 8-10 heterogeneous chunks, split the verification into batches to prevent the model from conflating evidence formats or missing cross-modal correlations.

This prompt template is designed for verification systems that must handle claims where evidence spans multiple modalities—text passages, tables, charts, and structured data records. Unlike text-only evidence matching, this prompt instructs the model to apply format-aware comparison logic: it must normalize table rows into claim-equivalent statements, interpret chart descriptions as data points, and reconcile structured fields with natural language assertions. The template is structured to produce a consistent JSON output that downstream systems can validate, log, and route for human review when confidence is low or evidence is missing.

textCopy
You are an evidence matching system that verifies claims against sources spanning text, tables, and structured data. Your task is to determine whether each claim is supported, contradicted, or unverifiable by the provided evidence, applying format-aware comparison logic.

## INPUT

Claims to verify:
[CLAIMS]

Evidence sources (may include text passages, table representations, and structured data records):
[EVIDENCE_SOURCES]

## FORMAT-AWARE COMPARISON RULES

1. **Text-to-Text Matching**: Compare claim assertions directly against textual evidence. Flag semantic equivalence, partial support, and direct contradiction.

2. **Table-to-Text Matching**: For tabular evidence, extract the relevant row and column values as a natural language statement before comparing to the claim. Example: a table row `| Q3 Revenue | $4.2M |` becomes "Q3 revenue was $4.2 million." Then compare this normalized statement to the claim.

3. **Structured Data-to-Text Matching**: For JSON, key-value pairs, or database records, convert the relevant fields into a declarative sentence before comparison. Example: `{"product": "WidgetX", "launch_date": "2024-03-15"}` becomes "WidgetX launched on March 15, 2024."

4. **Chart-to-Text Matching**: For chart descriptions or extracted chart data, identify the data series, axis values, and trend direction. Convert to a claim-equivalent statement before comparison. If only a chart description is provided (not raw data), note the interpretation layer in your confidence assessment.

5. **Numerical Comparisons**: When claims and evidence both contain numbers, apply a tolerance window of [NUMERICAL_TOLERANCE] for approximate matches. Flag exact matches, within-tolerance matches, and out-of-tolerance contradictions separately.

6. **Unit Normalization**: Before comparing numerical values, normalize units to a common standard. If units cannot be normalized (e.g., incompatible dimensions), flag as "incomparable" rather than supported or contradicted.

7. **Multi-Modality Corroboration**: When evidence from multiple formats supports the same claim, note the corroboration but do not double-count. When formats conflict (e.g., text says "increased" but table shows decrease), flag the contradiction with both sources cited.

## OUTPUT SCHEMA

Return a JSON object with the following structure:

```json
{
  "verification_results": [
    {
      "claim_id": "string - identifier from input claims",
      "claim_text": "string - the original claim text",
      "verdict": "supported | contradicted | partially_supported | unverifiable | incomparable",
      "evidence_matches": [
        {
          "source_id": "string - identifier from evidence sources",
          "source_type": "text | table | structured_data | chart_description",
          "relevant_excerpt": "string - the specific evidence text or normalized representation",
          "match_type": "direct_support | partial_support | contradiction | no_match",
          "normalization_applied": "string or null - describe any format normalization performed",
          "confidence": "high | medium | low",
          "explanation": "string - brief explanation of the match reasoning"
        }
      ],
      "overall_confidence": "high | medium | low",
      "missing_evidence_gap": "string or null - if unverifiable, describe what evidence would be needed",
      "cross_format_conflicts": [
        {
          "source_a_id": "string",
          "source_b_id": "string",
          "conflict_description": "string"
        }
      ]
    }
  ],
  "unmatched_claims": ["claim_ids with zero evidence matches"],
  "normalization_notes": ["string - any global notes about format normalization challenges"]
}

CONSTRAINTS

• Do not fabricate evidence. If no evidence addresses a claim, set verdict to "unverifiable" and populate missing_evidence_gap.
• For partial matches, explicitly identify which claim components are supported and which are not.
• When evidence sources conflict, report the conflict in cross_format_conflicts rather than silently selecting one source.
• If a table or structured data record is ambiguous, note the ambiguity in normalization_applied and reduce confidence.
• Do not treat chart descriptions as equivalent to raw chart data. Flag the interpretation gap.
• For numerical claims, always document the tolerance window used in the explanation.
• If [RISK_LEVEL] is "high", mark any medium or low confidence results for human review.

EXAMPLES

[FEW_SHOT_EXAMPLES]

RISK LEVEL

[RISK_LEVEL]

To adapt this template for your system, start by defining your [CLAIMS] input format. Claims should be pre-extracted and assigned unique identifiers before reaching this prompt—this template assumes clean claim objects, not raw text. For [EVIDENCE_SOURCES], structure each source with an id, type field (one of text, table, structured_data, chart_description), and the content itself. If your retrieval system returns tables as markdown or HTML, preserve that formatting; the model's format-aware rules depend on recognizing table structure. The [NUMERICAL_TOLERANCE] placeholder should be replaced with a domain-appropriate value—for financial claims, ±1% may be appropriate; for scientific claims, you may need tighter windows. The [FEW_SHOT_EXAMPLES] placeholder is critical for cross-modal matching: include at least one example each of table-to-claim matching, structured-data-to-claim matching, and a cross-format conflict case. Set [RISK_LEVEL] to "high" for regulated domains where human review is mandatory for anything below high confidence. After pasting this prompt, validate the JSON output against the schema before allowing it to flow into downstream systems—malformed JSON in verification pipelines can silently drop claims from audit trails.

The Evidence Matching Across Modalities prompt is not a standalone tool—it is a component inside a larger verification pipeline. The harness must handle format detection, evidence normalization, and output validation before the prompt result reaches a downstream consumer or a human reviewer. Because claims and evidence can arrive as plain text, markdown tables, JSON blobs, or chart descriptions, the application layer should pre-process inputs into a canonical representation before calling the model. This means extracting table rows into structured arrays, converting chart descriptions into labeled data points, and normalizing text claims into atomic, self-contained assertions. The prompt template expects [CLAIM] and [EVIDENCE] placeholders; the harness is responsible for populating them with format-annotated content so the model knows whether it is comparing text-to-table, table-to-text, or text-to-text.

Validation and retry logic. The model output must conform to a strict JSON schema that includes match_type (support/contradict/insufficient/partial), evidence_location (with format-specific pointers like row indices, column headers, or paragraph offsets), and comparison_logic (a human-readable explanation of how the cross-format comparison was performed). After each model call, run a schema validator. If the output fails validation, retry once with the validation error appended to the prompt as a [PREVIOUS_ERROR] block. If the second attempt also fails, route the claim-evidence pair to a human review queue with the raw model output and validation trace attached. Do not silently accept malformed outputs—cross-format evidence matching errors compound downstream when verification reports are generated.

Model selection and latency budgeting. Cross-modal reasoning benefits from models with strong structured data understanding. Prefer models that have demonstrated table-reading and chart-interpretation capabilities in your internal evals. For high-throughput pipelines, consider a two-tier architecture: a fast, cheaper model for clear text-to-text matches and a more capable model for table-to-text or chart-to-claim comparisons. The harness should inspect the evidence format at ingestion time and route accordingly. Log every routing decision, model choice, and latency measurement so you can tune thresholds over time.

Human review integration. When the model returns match_type: partial or match_type: insufficient, the harness should package the claim, evidence, model output, and source metadata into a structured review task. Include the comparison_logic field so reviewers can quickly assess whether the model's reasoning was sound. For regulated or high-stakes domains, require human sign-off on all contradict matches before they appear in verification reports. The harness should expose a review endpoint that accepts approve/override/reject decisions and logs them for audit.

Observability and eval hooks. Instrument the harness to log every prompt call with input hashes, output schemas, validation pass/fail status, retry count, and final disposition (auto-verified, escalated, errored). Run periodic eval sweeps using a golden dataset of cross-format claim-evidence pairs with known match types. Watch for format-specific failure modes: the model may correctly match text-to-text claims but hallucinate column mappings in table-to-text comparisons, or it may treat chart descriptions as literal text rather than visual data summaries. Your eval set should include equal representation of text, table, and chart-derived evidence to catch these skews before they reach production.