Custom Metric Definition Prompt Template

This prompt is a design tool for AI quality engineers, domain experts, and platform teams who need to define a new evaluation metric for a task where standard metrics like accuracy, BLEU, or generic helpfulness rubrics are insufficient. The job-to-be-done is producing a structured, repeatable scoring specification that captures domain-specific quality signals, handles edge cases explicitly, and can be calibrated against known good and bad outputs. You are not running an evaluation here; you are designing the measurement instrument that an LLM judge or code-based validator will later execute. The ideal user understands the task deeply enough to articulate what 'good' looks like in ways a general-purpose evaluator cannot, and needs to encode that judgment into a reusable artifact that survives team handoff and prompt version changes.

Use this prompt when you face a task where surface-level correctness masks deeper quality dimensions—for example, a customer support reply that is factually accurate but tone-deaf, a code review that catches syntax errors but misses architectural risks, or a summarization that is concise but omits critical safety information. The prompt template forces you to define the scoring function, specify edge-case handling rules, and provide calibration examples that anchor the metric to real outputs. It is particularly valuable when you need to align multiple stakeholders on what 'good' means before building an automated eval harness, or when you are migrating from spot-check human review to a systematic, model-graded evaluation pipeline. The output is a specification document, not a runtime prompt, so you should expect to iterate on it with domain experts before handing it off to an LLM judge prompt or a code-based validator.

Do not use this prompt when a standard metric already captures the quality dimension you care about—if BLEU or ROUGE suffices for your translation task, or if exact-match accuracy works for your classification problem, you do not need a custom metric. Avoid this prompt when you lack access to domain experts who can define edge cases and calibrate examples; the metric will only be as good as the judgment encoded into it. This prompt is also not a substitute for human evaluation in high-risk domains like healthcare or legal review—it produces a specification for an automated judge, but the judge itself must still be validated against human raters and subject to governance controls. Finally, do not use this prompt to define metrics for tasks you cannot provide concrete examples for; without calibration anchors, the resulting metric will drift and produce unreliable scores in production.

This prompt template is designed to help domain experts and AI quality engineers define a structured, repeatable evaluation metric for a specific task. Unlike generic rubrics, a custom metric definition captures the nuanced scoring logic, edge-case handling, and calibration examples that make an eval harness reliable. The output is a complete metric specification that can be handed directly to an LLM judge, used to guide human annotators, or implemented as a programmatic scoring function. Replace every square-bracket placeholder with your task-specific details before use.

textCopy
You are an expert evaluation designer. Your task is to define a precise, repeatable scoring metric for evaluating AI-generated outputs for the following task.

## TASK DESCRIPTION
[TASK_DESCRIPTION]

## METRIC NAME
[METRIC_NAME]

## WHAT THIS METRIC MEASURES
[MEASUREMENT_GOAL]

## SCORING SCALE
Define a [SCALE_TYPE] scale from [MIN_SCORE] to [MAX_SCORE]. For each score level, provide:
- The score value
- A clear, behavioral description of what that score represents
- At least one concrete example of an output that would receive that score

## EDGE CASES AND BOUNDARY RULES
Describe how to handle the following situations:
- [EDGE_CASE_1]
- [EDGE_CASE_2]
- [EDGE_CASE_3]

## CALIBRATION EXAMPLES
Provide [NUMBER] calibration examples. For each example, include:
- The input context
- The AI-generated output
- The correct score
- A brief justification referencing the scoring scale

## OUTPUT FORMAT
Return a JSON object with the following schema:
{
  "metric_name": "string",
  "measurement_goal": "string",
  "scale": {
    "type": "string (e.g., likert, binary, continuous)",
    "min": number,
    "max": number,
    "levels": [
      {
        "score": number,
        "label": "string",
        "description": "string",
        "example": "string"
      }
    ]
  },
  "edge_case_rules": [
    {
      "case": "string",
      "rule": "string"
    }
  ],
  "calibration_examples": [
    {
      "input_context": "string",
      "output": "string",
      "score": number,
      "justification": "string"
    }
  ]
}

## CONSTRAINTS
[CONSTRAINTS]

After pasting this template, the most critical step is populating the calibration examples. These examples anchor the metric in concrete behavior and are the single most effective defense against judge drift and inter-rater disagreement. Choose examples that sit near score boundaries, not just clear-cut cases. For high-stakes evaluation workflows, run the completed metric definition through a validation step: have two independent LLM judges or human raters score the same calibration examples using your new metric and measure their agreement. If agreement is below your threshold, refine the level descriptions and boundary rules before deploying the metric into your eval harness.

This prompt is not a one-off query; it is a specification generator that must be validated before it becomes part of your evaluation infrastructure. The output is a structured metric definition—scoring function, edge-case rules, calibration examples—that downstream eval runners will execute. Treat the generated spec as code: it needs schema validation, behavioral testing against known outputs, and version control before it gates any release decision.

Wire the prompt into a two-stage pipeline. Stage 1 (Generation): Send the prompt with the domain expert's task description, example good/bad outputs, and any existing rubric fragments. Collect the structured JSON output and validate it against a strict schema that requires at minimum metric_name, scoring_function (with clear logic or pseudocode), edge_cases (an array of scenarios and expected behavior), and calibration_examples (at least three scored pairs with justification). Reject and retry if the schema fails. Stage 2 (Verification): Run the generated metric against a small, held-out golden set of 10–20 output pairs that the domain expert has pre-scored. Compare the metric's scores to the expert's scores. If the correlation is below a threshold (e.g., Spearman's ρ < 0.8), flag for human review. Log the metric spec, the verification results, and the reviewer's decision to an audit trail. Use a model with strong instruction-following and structured output support (e.g., GPT-4o, Claude 3.5 Sonnet) and set temperature=0 for deterministic spec generation.

Do not deploy a generated metric directly to a release gate without this verification loop. The most common failure mode is a metric that looks plausible but mis-scores edge cases the domain expert would catch immediately. If the task is high-risk (e.g., healthcare, finance, legal), require human sign-off on the metric definition before it enters any automated eval pipeline. Store the approved metric spec in a version-controlled registry alongside the prompt version that generated it, so you can trace regressions in the metric itself back to changes in the definition prompt.