Success Metric Operationalization Prompt Template

This prompt is a planning pre-processor designed for agent architects and AI engineers who need to bridge the gap between human-readable success criteria and machine-evaluable metrics. Its core job is to take a set of qualitative success criteria—such as 'the system should respond quickly' or 'users should find the output useful'—and produce structured metric definitions that downstream evaluation, monitoring, and replanning modules can consume. The ideal user is someone building an agent planner or goal-clarification module who has already produced a set of success criteria but finds them too ambiguous for automated evaluation. Required context includes the original goal statement, the list of qualitative success criteria, any known constraints on measurement (e.g., no user surveys, limited telemetry), and the intended operational environment where these metrics will be collected.

Use this prompt when success criteria are expressed in natural language and need to be converted into concrete, instrumentable definitions with collection methods, threshold values, measurement cadence, and feasibility notes. It is particularly valuable when building autonomous agent systems that must self-evaluate their own outputs against pre-defined success criteria, or when setting up monitoring dashboards that track agent performance over time. The prompt expects you to provide a clear [GOAL] statement and a set of [SUCCESS_CRITERIA] expressed as natural language statements. It will return a structured output containing metric names, collection methods, threshold values, measurement frequency, and feasibility assessments. This output can be directly fed into evaluation harnesses, logging pipelines, or agent monitoring systems.

Do not use this prompt when success criteria are already fully instrumented with known telemetry and collection pipelines—in that case, you need a metric validation prompt, not a definition prompt. Do not use it when the goal is purely qualitative with no measurement intent, such as exploratory research or creative brainstorming where success is inherently subjective. Do not use it in regulated domains where metric definitions must be authored by a qualified human with domain expertise, such as clinical trial endpoints or financial audit criteria. Finally, do not use this prompt as a substitute for human judgment about what constitutes success—it operationalizes criteria you provide, but it cannot validate whether those criteria are the right ones for your business or user needs. The next step after generating metric definitions is to review them for feasibility, wire them into your evaluation harness, and run a calibration check to ensure the thresholds align with actual user expectations.

The following prompt template is designed to be pasted directly into your planning module or agent pre-processor. It takes a high-level goal and its associated qualitative success criteria and forces the model to output a rigorous, structured metric definition. The template uses square-bracket placeholders for all variable inputs. Before using it, ensure you have a clearly defined goal and a set of success criteria that need operationalization. This prompt is most effective when the qualitative criteria are already well-scoped; for ambiguous inputs, pair it with an Ambiguous Goal Refinement or Constraint Elicitation prompt first.

codeCopy
You are a measurement architect. Your task is to convert qualitative success criteria into a set of measurable, operational metrics that can be tracked by an automated system.

# GOAL
[GOAL_DESCRIPTION]

# QUALITATIVE SUCCESS CRITERIA
[SUCCESS_CRITERIA]

# CONSTRAINTS
- Available data sources: [DATA_SOURCES]
- Measurement cadence: [CADENCE]
- Risk tolerance for proxy metrics: [PROXY_RISK_TOLERANCE]

# OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "metrics": [
    {
      "metric_name": "string",
      "definition": "string (precise, formulaic)",
      "collection_method": "string (how to get the data)",
      "target_value": "number | string",
      "measurement_cadence": "string",
      "proxy_quality_assessment": "string (direct, strong proxy, weak proxy)",
      "feasibility_notes": "string (blockers, data gaps, assumptions)"
    }
  ],
  "coverage_assessment": "string (explain which criteria are fully, partially, or not covered by the metrics)",
  "recommended_leading_indicators": ["string"],
  "recommended_lagging_indicators": ["string"]
}

# INSTRUCTIONS
1. For each qualitative criterion, derive one or more quantitative metrics.
2. If a direct metric is impossible, propose the strongest possible proxy and explicitly flag it as a proxy.
3. For every metric, specify a concrete target value or threshold. If you cannot determine one, state what information is needed to set it.
4. Identify any feasibility blockers, such as missing data sources or instrumentation gaps.
5. Distinguish between leading indicators (predictive) and lagging indicators (outcome-based).

To adapt this template, replace the placeholders with your specific context. [GOAL_DESCRIPTION] should be a concise statement of the objective. [SUCCESS_CRITERIA] should be a list of the qualitative 'Definition of Done' statements you need to measure. [DATA_SOURCES] is critical for feasibility; list the actual logs, databases, or APIs available. [PROXY_RISK_TOLERANCE] should be set to 'low' for high-stakes decisions where a weak proxy could be misleading, or 'moderate' for exploratory tracking. After running the prompt, always validate the output against the OUTPUT_SCHEMA and manually review the proxy_quality_assessment for any metric flagged as a 'weak proxy' before integrating it into a dashboard or automated decision loop.

This prompt is designed to sit between a goal clarification module and a plan generation engine. It takes a refined goal and its qualitative success criteria as input and produces a structured, machine-readable metric specification. The output is not a final dashboard or monitoring query; it is a definition that downstream tools, evaluation harnesses, and human reviewers can use to build actual measurement. Wire this prompt as a synchronous step in a planning pre-processor, not as a standalone chat interaction. The model choice matters: use a model with strong structured output and reasoning capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) because the task requires both creative operationalization and strict schema adherence.

Implement a validation layer immediately after the model response. The output must conform to a strict JSON schema that includes required fields for each metric: metric_name, definition, collection_method, unit, threshold_value, measurement_cadence, proxy_quality_assessment, and feasibility_notes. If validation fails, retry once with the validation errors injected into the [CONSTRAINTS] placeholder. If the second attempt also fails, route the output to a human review queue with the original goal, the failed metric definitions, and the specific schema violations flagged. Log every attempt, including the raw prompt, model response, validation result, and retry count, for observability and prompt debugging. For high-stakes domains (healthcare, finance, safety-critical systems), always require human approval on the final metric specification before it feeds into downstream planning or execution.

The prompt's [FEASIBILITY_CONSTRAINTS] placeholder is critical for production use. Populate it dynamically from your system's known capabilities: available data sources, logging infrastructure, metric collection tooling, and team bandwidth. For example, if your system cannot support real-time metric collection, inject that constraint to prevent the model from proposing streaming metrics. If you have a metrics store like Prometheus or a data warehouse, list the available aggregation windows and query capabilities. This grounding prevents the model from producing elegant but unimplementable metric definitions. After the prompt runs, consider a secondary validation step that checks whether each proposed collection_method references only the tools and data sources listed in the feasibility constraints. Flag any metric that requires unavailable infrastructure for human review.

Integrate this prompt's output with your evaluation framework. The generated metrics become the basis for LLM-judge rubrics, regression test pass/fail criteria, and production monitoring alerts. Store the metric definitions alongside the plan they were generated for, versioned together. When a plan is replayed or adapted for a similar goal, reuse the metric specification as a starting point rather than regenerating from scratch. Avoid the common failure mode of treating metric operationalization as a one-time step: as goals evolve and constraints change, the metrics must be re-validated. Schedule a periodic review of active metric definitions against current system capabilities and goal alignment.