Reference-Guided Precision and Recall Scoring Prompt Template

This prompt is for evaluation leads and AI engineers who need to move beyond 'looks good to me' when comparing a model's output against a known-correct reference answer. The job-to-be-done is automated, structured measurement of how much of the reference content the model captured (recall) and how much of the model's output was actually correct (precision), producing standard IR metrics adapted for LLM-generated text. The ideal user is someone maintaining a golden dataset of reference answers and building a regression gate that must fail a deployment when factual completeness or accuracy drops below a defined threshold. Required context includes the model's raw output, the reference answer, and an optional unit of analysis specification (token-level, fact-level, or entity-level).

Do not use this prompt when the reference answer is one of many valid interpretations, when the task is open-ended creative generation, or when you need a single holistic quality score rather than decomposed precision and recall metrics. This prompt is also inappropriate for evaluating outputs where the reference is a policy document or style guide rather than a factual answer—use a rubric-based or pairwise comparison prompt instead. If your evaluation requires numerical tolerance awareness (e.g., 'within 5% of 42.7'), use the Numerical Tolerance Evaluation prompt. If you need to detect contradictions specifically, pair this with the Contradiction Detection prompt rather than relying on precision/recall alone to catch safety-critical conflicts.

Before implementing this prompt, ensure you have a stable set of reference answers that represent ground truth for your domain. The prompt assumes the reference is authoritative and complete. If your reference answers are incomplete, your recall scores will be misleadingly high and your precision scores will penalize the model for adding correct information the reference missed. Start with a small calibration set of 20-30 examples where you've manually computed precision and recall to validate that the LLM judge's scores align with human judgment before scaling to your full regression suite.

This template provides the core instruction set for an LLM judge tasked with evaluating a model's output against a known-correct reference. It is designed to be dropped directly into your evaluation harness. The prompt instructs the judge to decompose both the reference and the candidate output into discrete, checkable units—either tokens or atomic facts—before computing standard information retrieval metrics. Use this when you have a golden dataset and need automated, repeatable scoring that goes beyond simple string matching to capture semantic equivalence.

textCopy
You are an evaluation judge. Your task is to compare a model's output against a reference answer and compute precision, recall, and F1 scores.

# INPUTS
## Reference Answer (Ground Truth)
[REFERENCE]

## Model Output to Evaluate
[CANDIDATE]

## Decomposition Strategy
[STRATEGY: token | fact | entity]

## Output Schema
You must respond with a single JSON object conforming to this schema:
{
  "precision": <float 0.0-1.0>,
  "recall": <float 0.0-1.0>,
  "f1_score": <float 0.0-1.0>,
  "reference_units": ["<extracted unit 1>", "<extracted unit 2>", ...],
  "candidate_units": ["<extracted unit 1>", "<extracted unit 2>", ...],
  "matched_units": ["<unit matched in both>", ...],
  "missed_units": ["<unit in reference but not in candidate>", ...],
  "extraneous_units": ["<unit in candidate but not in reference>", ...],
  "justification": "<brief explanation of the scoring decision>"
}

# CONSTRAINTS
- Decompose the reference and candidate into discrete units based on the specified [STRATEGY].
- For 'token' strategy: split on whitespace and punctuation, normalize case.
- For 'fact' strategy: extract standalone, verifiable atomic claims.
- For 'entity' strategy: extract named entities (people, places, organizations, dates, quantities).
- A candidate unit matches a reference unit if they are semantically equivalent, not just string-identical.
- Precision = (number of matched units) / (total number of candidate units).
- Recall = (number of matched units) / (total number of reference units).
- F1 = 2 * (Precision * Recall) / (Precision + Recall). If both are 0, F1 is 0.
- Do not include any text outside the JSON object.

To adapt this template, replace the [REFERENCE] and [CANDIDATE] placeholders with your data at runtime. The [STRATEGY] placeholder should be set to token, fact, or entity depending on the granularity you need. For most knowledge-work evaluations, fact provides the best signal-to-noise ratio. For code or structured data, token may be more appropriate. The output schema is strict JSON, which allows your harness to parse the score and the detailed unit lists directly. The justification field is critical for debugging scoring disagreements and should be logged alongside the metrics. Before deploying this into a production gate, run it against a calibration set of 20-50 examples scored by a human to verify that the LLM judge's precision and recall values align with your team's expectations for semantic equivalence.

The Reference-Guided Precision and Recall Scoring Prompt Template is designed to be called programmatically as a scoring function within an evaluation harness, not as a one-off chat interaction. The prompt expects a model output, a reference answer, and a scoring configuration (token-level or fact-level alignment, threshold preferences). The harness is responsible for assembling these inputs, invoking the LLM judge, parsing the structured score response, and routing the result to your evaluation database or CI/CD gate. Treat this prompt as a deterministic scorer: same inputs should produce consistent precision, recall, and F1 scores, though minor LLM variance is expected. Log the raw judge response alongside the parsed scores for auditability.

Validation and Retry Logic: The prompt instructs the judge to return a JSON object with precision, recall, f1, and alignment_details fields. Your harness must validate this schema before accepting the score. If the response fails JSON parsing or is missing required fields, implement a retry with the same prompt but append the malformed response and a terse correction instruction: Your previous response was not valid JSON or was missing required fields. Return ONLY the JSON object with precision, recall, f1, and alignment_details. Set a maximum of 2 retries. If validation still fails after retries, log the failure, mark the evaluation as SCORING_ERROR, and escalate for human review rather than silently assigning a default score. For high-stakes evaluation pipelines (regression gates, model release decisions), require a human to spot-check a random sample of scored pairs, especially those near your F1 threshold boundary.

Model Choice and Configuration: Use a model with strong instruction-following and structured output capabilities for the judge role. Temperature should be set low (0.0–0.2) to maximize scoring consistency across repeated runs. If your evaluation platform supports it, pin the judge model version to avoid score drift from silent model updates. For cost-sensitive pipelines running thousands of evaluations, consider using a smaller, fine-tuned judge model that has been calibrated against human precision/recall annotations on your domain data. The prompt template includes a [CONSTRAINTS] placeholder where you can inject domain-specific scoring rules, such as 'Treat synonyms as matches' or 'Penalize extra information that contradicts the reference.'

Integration with Evaluation Pipelines: Wire this prompt into your existing evaluation framework (e.g., Braintrust, LangSmith, custom eval harness) as a scorer function. The scorer should accept (output, reference, config) and return {precision, recall, f1, alignment_details, raw_response}. Store all scores in your evaluation database with metadata: prompt version, judge model version, timestamp, and the raw judge response. For CI/CD gating, define thresholds in your pipeline configuration—for example, f1 >= 0.8 for automated pass, f1 < 0.6 for automated fail, and 0.6 <= f1 < 0.8 for human review. These thresholds should be calibrated against your domain's tolerance for false positives versus false negatives. Do not use the same threshold for a chatbot's factual accuracy as you would for a medical coding system.

Failure Modes to Monitor: The most common production failure is the judge hallucinating alignment spans that don't exist in either text. Mitigate this by including span-grounding instructions in the prompt (already present in the template) and by periodically auditing alignment_details against the source texts. A second failure mode is score inflation on short outputs where precision appears perfect because the model said very little. Consider adding a coverage or completeness check alongside precision/recall for outputs that are suspiciously brief. A third failure mode is judge bias toward the reference's phrasing style rather than semantic content. If you observe this, add explicit instructions in [CONSTRAINTS] to prioritize meaning over surface-form matching. Log all three metrics (precision, recall, F1) separately—a high F1 can mask a precision-recall trade-off that matters for your use case.