Reference-Free Evaluation Prompt Template

This prompt is designed for AI quality engineers and platform teams who need to evaluate open-ended model outputs when a golden reference answer does not exist. Use it inside an automated eval harness to score outputs on coherence, relevance, factual consistency, safety, and instruction adherence. It is appropriate for summarization, creative generation, dialogue, and free-form analysis tasks where reference-based metrics like BLEU or ROUGE are insufficient. The prompt acts as an LLM judge that produces structured scores and justifications, enabling programmatic pass/fail gating in CI/CD pipelines.

To implement this correctly, you must provide the original user request as [INPUT], the model's response as [OUTPUT], and any relevant context such as retrieved documents or system instructions as [CONTEXT]. The prompt will return a JSON object with per-axis scores, an aggregate score, and a justification for each rating. Wire the output into a validation step that checks for schema conformance, score ranges, and justification presence before accepting the result. For high-stakes domains, always pair this automated evaluation with a human review sample and log disagreements to detect judge drift over time.

Do not use this prompt when a verified reference answer is available; use the Reference-Based Evaluation Prompt Template instead. Do not use it as a substitute for human review in high-risk domains such as clinical or legal workflows without additional oversight. Before deploying, calibrate the judge against a set of human-annotated examples to establish baseline agreement rates and tune your pass/fail thresholds.

This template provides a reusable structure for a reference-free evaluation judge. It instructs the model to act as a quality assessor, scoring a generated output across multiple intrinsic dimensions such as coherence, relevance, and safety. The prompt is designed to be dropped into an eval harness, with square-bracket placeholders that you replace with your specific task context, the output to be evaluated, and your quality rubric. It includes built-in self-consistency checks and hallucination guardrails to produce a structured, machine-readable score report.

textCopy
You are a strict quality assurance evaluator. Your task is to assess a single AI-generated output against a provided rubric. You do not have access to a ground-truth reference. You must evaluate the output based solely on its intrinsic qualities and its adherence to the provided context and constraints.

# EVALUATION CONTEXT
[CONTEXT]

# OUTPUT TO EVALUATE
[OUTPUT]

# QUALITY RUBRIC
Evaluate the output on the following dimensions. For each, provide a score from 1 (worst) to 5 (best) and a concise, evidence-based justification.

[RUBRIC]

# SELF-CONSISTENCY AND HALLUCINATION CHECK
1. Factual Claims: List every discrete factual claim made in the output.
2. Source Check: For each claim, determine if it is directly supported by the [CONTEXT]. Mark each as 'Supported', 'Unsupported', or 'Contradicted'.
3. Internal Logic: Check if any statements in the output logically contradict each other. If so, explain the contradiction.

# OUTPUT FORMAT
You must respond with a single, valid JSON object. Do not include any text outside the JSON object. The JSON object must conform to this schema:
{
  "overall_score": <float between 1.0 and 5.0>,
  "overall_justification": "<summary justification>",
  "dimension_scores": [
    {
      "name": "<dimension name>",
      "score": <integer>,
      "justification": "<evidence-based justification>"
    }
  ],
  "hallucination_report": {
    "claims": [
      {
        "claim": "<text of the claim>",
        "verdict": "<Supported|Unsupported|Contradicted>",
        "evidence": "<relevant snippet from context or 'None'>"
      }
    ],
    "internal_contradictions": ["<description of contradiction, or null>"],
    "hallucination_severity": "<None|Low|Medium|High|Critical>"
  }
}

To adapt this template, replace the placeholders with your specific task details. The [CONTEXT] should contain the original user query, retrieved documents, or any other grounding information. The [OUTPUT] is the text you are evaluating. The [RUBRIC] is the most critical part: define 3-5 clear, independent dimensions (e.g., 'Coherence', 'Relevance to Query', 'Tone Appropriateness') with descriptions of what a score of 1 and 5 look like for each. After running the prompt, always validate the output against the JSON schema. If the model fails to produce valid JSON, implement a retry with a stronger format constraint or use a repair prompt. For high-stakes evaluations, the hallucination report should be reviewed by a human to verify the severity classification before the score is accepted as a release gate.

The reference-free evaluation prompt is not a one-off quality check; it is a scoring function that must be integrated into a deterministic harness to be useful for regression testing. The harness is responsible for calling the model, parsing the structured score, validating the output against a schema, and routing the result to a logging or gating system. Without this harness, the evaluation is just an opinion. With it, the evaluation becomes a repeatable measurement that can block a bad deployment or trigger a human review.

Start by wrapping the prompt in a function that accepts the [OUTPUT_TO_EVALUATE] and [EVALUATION_CRITERIA] as arguments. The function should call the model with temperature=0 and a strict JSON mode or structured output constraint to enforce the [OUTPUT_SCHEMA]. After the call, validate the response: check that every required field (overall_score, coherence_score, relevance_score, safety_flags) is present and within expected ranges. If validation fails, retry once with an explicit error message injected into the prompt context. If it fails again, log the raw output and escalate for manual review. For high-stakes domains, always route safety_flags with a severity of high to a human approval queue before the score is accepted.

To make this production-grade, log every evaluation result as a structured event with the prompt version, model ID, timestamp, and the raw scores. This log becomes your audit trail for release gates. When wiring this into a CI/CD pipeline, treat the overall_score as the primary signal: set a pass threshold (e.g., >= 0.8) based on historical score distributions from your golden dataset. If the score drops below the threshold on a new prompt version, fail the build. Avoid using this prompt as the sole gate for factuality; pair it with a reference-based or hallucination detection eval when ground-truth evidence is available. The reference-free eval measures intrinsic quality, not correctness.