Preference Selection Prompt for Tool-Use Sequences

This prompt is designed for agent platform teams who need to make a relative quality judgment between two tool-call sequences. The core job-to-be-done is not to assign an absolute score, but to force a preference decision based on concrete, observable criteria: tool selection correctness, argument accuracy, call ordering logic, and appropriate abstention. The ideal user is an ML engineer, an evaluation lead, or a developer running a champion-challenger test, tuning agent behavior, or curating preference data for fine-tuning. You should use this prompt when you have two complete execution traces from different models, configurations, or prompt versions and you need to know which one is better for a specific task.

The prompt is most effective when integrated into a structured comparison harness. For example, you might be comparing a new gpt-4o agent configuration against a production claude-sonnet-4 baseline on a set of 200 customer service tasks. Instead of manually reviewing logs, you feed each pair of tool-call sequences into this prompt. The output is a structured JSON verdict that tells you which sequence won, why it won, and a confidence score. This data can then be aggregated to produce a statistically significant win/loss/tie report. The prompt requires both sequences to be provided with full argument payloads and execution context; it cannot infer what a tool does from its name alone.

Do not use this prompt when you need an absolute quality score for a single sequence, as it is calibrated for relative comparison. It is also unsuitable if only one sequence is available, if the task does not involve tool calls, or if the sequences are incomplete. Avoid using it as a standalone quality gate without human calibration; always validate a sample of preference judgments against human expert ratings to detect judge bias or systematic errors. The next step after reading this section is to review the prompt template and understand the exact input schema required to produce a reliable preference verdict.

This prompt template is designed for agent platform teams who need to evaluate and compare tool-call sequences from different models or configurations. It produces a structured preference verdict based on tool selection correctness, argument accuracy, call ordering, and appropriate abstention. The template uses square-bracket placeholders that you replace with real data before sending it to the judge model. This prompt is particularly useful for champion-challenger testing, RLHF data pipeline generation, and model selection workflows where relative quality of tool-use matters more than absolute scores.

textCopy
You are an expert evaluator of AI agent tool-use sequences. Your task is to compare two tool-call traces and select the preferred one based on correctness, efficiency, and safety.

## Evaluation Criteria (in priority order)
1. **Tool Selection Correctness**: Did the agent choose the right tools for the task? Incorrect tool selection is a critical failure.
2. **Argument Accuracy**: Are the arguments passed to each tool correct, complete, and well-typed? Missing required arguments or hallucinated values are serious errors.
3. **Call Ordering**: Is the sequence of tool calls logical and efficient? Unnecessary calls, redundant operations, or incorrect ordering reduce preference.
4. **Appropriate Abstention**: Did the agent correctly abstain from calling tools when it lacked sufficient information or when no tool was appropriate? Unnecessary tool calls are worse than correct abstention.
5. **Error Handling**: If errors occurred, did the agent handle them gracefully with retries, fallbacks, or appropriate escalation?

## Inputs
**User Request**: [USER_REQUEST]

**Available Tools** (name, description, parameters):
[AVAILABLE_TOOLS]

**Execution Context** (system state, previous turns, constraints):
[EXECUTION_CONTEXT]

**Trace A** (model: [MODEL_A_ID]):
[TRACE_A]

**Trace B** (model: [MODEL_B_ID]):
[TRACE_B]

## Output Format
Return a JSON object with the following structure:
{
  "preferred_trace": "A" | "B" | "TIE",
  "confidence": 0.0-1.0,
  "justification": {
    "tool_selection": "Explanation of which trace had better tool selection",
    "argument_accuracy": "Explanation of argument quality differences",
    "call_ordering": "Explanation of sequence efficiency differences",
    "abstention_handling": "Explanation of abstention appropriateness",
    "error_handling": "Explanation of error recovery quality"
  },
  "key_differentiators": ["List of 1-3 specific factors that determined the preference"],
  "critical_failures": {
    "trace_a": ["List of critical failures in Trace A, if any"],
    "trace_b": ["List of critical failures in Trace B, if any"]
  }
}

## Constraints
- If both traces have critical failures, you may still express a preference based on severity, but confidence should be low.
- If both traces are functionally identical, return "TIE" with confidence reflecting your certainty about the tie.
- Do not let trace length or verbosity influence your judgment; focus on correctness and efficiency.
- If one trace correctly abstains and the other makes incorrect tool calls, strongly prefer the abstaining trace.
- Ignore model identity; evaluate only the traces as presented.

## Evaluation
Compare the two traces against the user request, available tools, and execution context. Provide your structured assessment.

To adapt this template for your evaluation pipeline, replace each square-bracket placeholder with real data before sending it to the judge model. The [USER_REQUEST] should contain the original task or query that triggered the tool-use sequence. The [AVAILABLE_TOOLS] placeholder expects a structured description of each tool's name, purpose, and parameter schema—this is critical for the judge to assess whether the correct tools were selected. The [EXECUTION_CONTEXT] should include any relevant system state, conversation history, or environmental constraints that affect tool selection decisions. For the traces, include the complete sequence of tool calls with arguments and return values for each model being compared. The [MODEL_A_ID] and [MODEL_B_ID] placeholders help with logging but should not influence the judge's decision. After adapting the template, validate that your tool descriptions are complete enough for the judge to make accurate assessments—missing parameter documentation will lead to unreliable preference labels. For high-stakes evaluation pipelines, run this prompt with multiple judge models and compare agreement rates before trusting the output.

This prompt is designed to sit inside an automated evaluation harness that compares tool-use sequences from two different agent configurations. The harness must supply the full execution trace—including tool names, arguments, call order, and any error responses—for both sequences. Before calling the LLM judge, validate that both traces are parseable and contain at least one tool call. If either trace is empty or malformed, skip the comparison and log a data-quality failure. The prompt expects structured input fields for [TRACE_A] and [TRACE_B], each containing a JSON array of tool-call objects with tool_name, arguments, timestamp, and result keys. Position bias is a known failure mode in pairwise comparison, so randomize whether Trace A or Trace B appears first in the prompt on each evaluation run and record the assignment for later analysis.

Wire the prompt into your evaluation pipeline as a post-execution step. After both agent configurations have completed the same task, capture their full tool-call sequences and pass them to this prompt. Parse the JSON output to extract the preferred_trace field (A, B, or tie), the confidence score, and the structured reasons object. Implement a validation layer that rejects malformed JSON, missing required fields, or confidence scores outside the 0-1 range. For high-stakes comparisons—such as champion-challenger model testing or RLHF data generation—add a second-pass verification: if confidence is below 0.7 or the verdict is tie, route the pair to a human reviewer or a more capable judge model. Log every comparison result with the randomized position assignment, both traces, the judge's verdict, and the confidence score. This log becomes your audit trail for detecting judge drift, position bias, or systematic preference patterns over time.

Model choice matters for this workflow. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models that may struggle with the multi-dimensional reasoning required to compare tool-call sequences across correctness, argument accuracy, ordering, and abstention simultaneously. Set temperature=0 for deterministic verdicts in automated pipelines. If you need calibrated confidence scores, run a calibration study comparing the judge's confidence against human preference agreement on at least 50 labeled pairs before trusting the scores in production. Implement a retry strategy for transient failures: if the output fails JSON parsing, retry once with the same input. If it fails twice, log the failure and escalate. Never silently accept an unparseable preference verdict—downstream systems that consume these labels will propagate the error into training data or model selection decisions.