Claim Extraction and Evidence Matching Trace Prompt

This prompt is designed for evaluation dataset builders and RAG quality engineers who need to construct a structured claim-evidence mapping from a single production trace. The task is to decompose a generated answer into atomic, verifiable claims and then attempt to match each claim to a specific retrieved passage captured in the trace. The output is a table that links each claim to its supporting evidence, a match confidence score, and a pointer to the trace span. Use this prompt when you are building a golden dataset for hallucination detection, calibrating an LLM judge, or auditing a RAG system's grounding behavior. It assumes you already have a trace containing the user query, the retrieved context chunks with their trace span IDs, and the final generated answer.

This prompt is not a real-time guardrail; it is a batch analysis tool for offline quality engineering. Do not use it in a synchronous request path where latency matters, as the decomposition and matching process is token-intensive and requires careful review. It is also not a substitute for human annotation when building high-stakes evaluation sets—treat its output as a first pass that requires human verification, especially for claims with low match confidence scores or complex multi-hop reasoning. The prompt works best with traces that have clean, attributable span IDs and retrieved passages that are self-contained enough to evaluate independently. If your traces lack span-level granularity or your retrieved context is heavily truncated, the evidence matching step will produce unreliable results.

Before running this prompt at scale, validate its output against a small set of manually annotated traces to calibrate your confidence thresholds. Pay special attention to claims that the model marks as 'supported' with low confidence—these are often cases where the retrieved passage is semantically related but does not actually entail the claim. After generating the claim-evidence table, use it to populate an evaluation dataset, feed it into a hallucination classifier, or identify systematic retrieval gaps. If you find that more than 20% of claims cannot be matched to any retrieved passage, investigate whether your retrieval pipeline is failing, your chunking strategy is too coarse, or the knowledge base itself has coverage gaps.

This template is the core instruction set for building an evaluation dataset. It takes a generated answer and the full retrieval trace as input, then decomposes the answer into discrete, verifiable claims. For each claim, it attempts to locate the exact supporting passage from the retrieved context, providing a match confidence score and a precise trace span pointer. Use this prompt when you need to quantify grounding at a granular level, not just a binary faithful/unfaithful label.

textCopy
You are an expert trace auditor. Your task is to extract every atomic, factual claim from a generated answer and match each claim to a specific retrieved passage in the provided trace.

## INPUT
[ANSWER]
[TRACE_DATA]

## OUTPUT_SCHEMA
Return a single JSON object with the key "claim_evidence_map", which is an array of objects. Each object must have the following fields:
- "claim_id": string (unique identifier, e.g., "C1")
- "claim_text": string (the exact atomic claim from the answer)
- "match_status": "MATCHED" | "UNSUPPORTED" | "CONTRADICTED"
- "matched_passage_text": string | null (the verbatim text from the trace that supports the claim)
- "trace_span_id": string | null (the unique identifier for the trace span containing the passage)
- "confidence_score": number (0.0 to 1.0, indicating how well the passage supports the claim)
- "reasoning": string (brief explanation of the match status and score)

## CONSTRAINTS
- Decompose the answer into the smallest possible factual claims. Do not group multiple facts into one claim.
- A claim is "UNSUPPORTED" if no retrieved passage provides evidence for it.
- A claim is "CONTRADICTED" if a retrieved passage directly refutes it.
- The "matched_passage_text" must be a direct quote from the [TRACE_DATA].
- The "trace_span_id" must be the exact identifier from the trace, not an invented one.
- If a claim is UNSUPPORTED or CONTRADICTED, set "matched_passage_text" and "trace_span_id" to null.
- Do not hallucinate trace spans or passage text.

## EXAMPLES
[EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

To adapt this template, replace the placeholders with your specific data. [ANSWER] should contain the full text of the model's generated response. [TRACE_DATA] must be a structured representation of the retrieval step, including the text of each retrieved chunk and its unique span ID. The [EXAMPLES] placeholder is critical for performance; provide 2-3 few-shot examples that demonstrate the difference between a MATCHED, UNSUPPORTED, and CONTRADICTED claim, especially showing how to handle partial matches. Set [RISK_LEVEL] to "HIGH" to trigger stricter confidence thresholds and more conservative matching in the model's internal reasoning. After generation, always validate the output JSON against the schema and verify that every trace_span_id in a MATCHED claim exists in the input [TRACE_DATA].

This prompt is designed to be a component in an offline evaluation pipeline, not a real-time user-facing feature. The primary integration point is a batch processing job that reads production traces—each containing a generated answer, the retrieved context, and trace span metadata—and produces a structured claim-evidence mapping table. The output is consumed by evaluation dashboards, dataset builders, and regression testing frameworks that compare grounding quality across prompt versions or model upgrades.

Wire the prompt into a Python or Node.js worker that iterates over serialized trace objects. For each trace, construct the prompt by injecting the generated answer into [GENERATED_ANSWER], the full list of retrieved passages with their trace span IDs into [RETRIEVED_PASSAGES], and any domain-specific taxonomy or constraint notes into [DOMAIN_CONTEXT]. Set [CONFIDENCE_THRESHOLD] to a float between 0.0 and 1.0—typically 0.7 for initial runs—to filter low-confidence matches. The model should be instructed to return a strict JSON array of claim objects. Validate the response immediately: check that claim_id values are unique, that every matched_passage_id exists in the input passage list, that match_confidence is a float, and that trace_span_id is present for matched claims. If validation fails, retry once with the validation error appended to the prompt as additional context. Log all validation failures and retries to your observability platform with the trace ID attached.

For high-stakes evaluation datasets—such as those used to gate a model upgrade or certify a RAG pipeline for regulated use—add a human review step. After the prompt produces the claim-evidence mapping, route claims with match_confidence below 0.9 or with match_status set to conflict or partial to a review queue. Reviewers should confirm or correct the mapping before the record enters the golden dataset. Store the final mapping alongside the original trace in your evaluation database, indexed by trace ID and prompt version. Avoid running this prompt on streaming or synchronous user request paths; the extraction and matching step adds latency and token cost that belongs in the evaluation layer, not the serving layer.