Tie-Breaking Prompt with Explicit Criteria

This prompt is a secondary resolver designed for evaluation pipelines where a standard pairwise comparison judge returns a tie or a confidence score below a defined threshold. Instead of accepting a draw, it forces a structured, auditable choice by applying a configurable hierarchy of tie-breaking dimensions. The primary job-to-be-done is automated model selection, RLHF data pipeline curation, or champion-challenger promotion when your production system requires a definitive winner and cannot tolerate an undecided outcome. The ideal user is an ML engineer or evaluation platform developer who already has two candidate outputs, the original task context, and a set of weighted criteria that reflect business priorities such as safety, conciseness, or instruction adherence.

Do not use this prompt as a replacement for a primary quality judge. It is a downstream resolver that activates only when the primary judge cannot discriminate. It assumes the two outputs are already deemed comparable in overall quality. If the outputs differ significantly in obvious ways, a standard pairwise comparison prompt is more appropriate. The prompt requires you to define explicit tie-breaking dimensions with weights before invocation. For example, you might configure safety: 0.4, instruction_adherence: 0.3, conciseness: 0.2, readability: 0.1. The resolver applies these weights in sequence, documenting which dimension broke the tie and why. This design makes the decision auditable and allows you to tune the hierarchy as your product priorities evolve.

Before deploying this prompt, establish a clear activation threshold from your primary judge—such as a confidence score below 0.7 or an explicit tie verdict. Wire the resolver into your pipeline as a conditional step, not a default path. Log every tie-breaking decision with the input context, the weighted criteria used, the winning output, and the dimension that broke the tie. This audit trail is essential for detecting criterion drift, identifying cases where the hierarchy produces counterintuitive results, and recalibrating weights over time. If the resolver itself cannot break the tie after exhausting all dimensions, escalate to human review rather than forcing an arbitrary choice.

This template acts as a forced-choice arbiter. It should only be invoked after a primary comparison prompt has returned a tie or a confidence score below your operational threshold. The prompt is designed to break the deadlock by applying explicit, weighted criteria that reflect your production priorities—such as safety, instruction adherence, or conciseness—rather than asking for a general preference. The square-bracket placeholders allow you to inject the two tied outputs, the original task context, and your specific tie-breaking dimensions without modifying the core logic.

textCopy
You are a tie-breaking evaluation judge. Your task is to choose between two outputs that a primary judge could not distinguish. You must produce a forced choice based on the weighted criteria provided. Do not return a tie.

## ORIGINAL TASK CONTEXT
[ORIGINAL_PROMPT]

## OUTPUT A
[OUTPUT_A]

## OUTPUT B
[OUTPUT_B]

## TIE-BREAKING CRITERIA
You must evaluate the two outputs against the following dimensions. Each dimension has a weight indicating its importance. A higher weight means the dimension is more critical for breaking the tie.

[CRITERIA]

## INSTRUCTIONS
1. For each criterion, briefly state which output is better and why. Be specific, referencing the output content.
2. Calculate a weighted score for each output by summing the wins per criterion multiplied by its weight.
3. Select the output with the higher total weighted score as the winner.
4. If the weighted scores are exactly equal, apply the escalation rule defined below.

## ESCALATION RULE
[ESCALATION_RULE]

## OUTPUT FORMAT
Return a single JSON object with the following schema:
{
  "winner": "A" | "B",
  "criteria_analysis": [
    {
      "criterion": "string",
      "weight": number,
      "winner": "A" | "B",
      "rationale": "string"
    }
  ],
  "weighted_score_a": number,
  "weighted_score_b": number,
  "escalation_triggered": boolean,
  "final_justification": "string"
}

To adapt this template, replace the [CRITERIA] placeholder with a structured list of your tie-breaking dimensions and their weights. For example, a safety-critical application might use: 1. Safety (weight: 10): Which output avoids harmful, biased, or disallowed content? 2. Instruction Adherence (weight: 8): Which output follows all explicit constraints from the original prompt? 3. Conciseness (weight: 3): Which output is more direct and avoids unnecessary verbosity?. The [ESCALATION_RULE] placeholder should define what happens in a true tie after weighting, such as "Default to Output A" or "Flag for human review." Always validate that the final JSON output matches the expected schema before accepting the verdict. In high-stakes domains like healthcare or finance, log the full criteria analysis and escalate true ties to a human reviewer instead of accepting a forced default.

The tie-breaking prompt is designed to sit behind an API endpoint or evaluation worker that receives two outputs and a set of weighted criteria. It should only be invoked when a primary pairwise comparison prompt returns a tie or a confidence score below a configured threshold. This keeps the tie-breaker as a specialized, cost-conscious step rather than running it on every comparison pair. The harness must supply the two candidate outputs, the original input or context that produced them, and the ordered list of tie-breaking dimensions with their associated weights. Without explicit weights, the prompt will default to equal weighting, which defeats the purpose of a configurable tie-breaker.

Validation and retry logic is critical because a malformed tie-breaker response can silently corrupt preference datasets. The harness must validate that the response contains exactly one preferred output identifier, a justification string, and a confidence score between 0 and 1. If the model returns both outputs as preferred, fails to select one, or produces a confidence below a minimum threshold (typically 0.6), the harness should retry once with an explicit instruction to force a choice. After two failures, the pair should be escalated to a human review queue with the full context, both outputs, and the model's last attempted justification. Log every tie-breaking decision with the criterion weights used, the model version, the retry count, and the final disposition (model-decided, escalated, or true-tie). This audit trail is essential for debugging judge drift and weight sensitivity over time.

Model choice matters for tie-breaking. Use a model with strong instruction-following and reasoning capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 2.0 Flash are appropriate. Avoid smaller or faster models that may default to positional bias or fail to apply weighted criteria consistently. Set temperature to 0 or near-zero to maximize determinism. If your pipeline processes high volumes, consider batching tie-breaker calls but never batch pairs that share the same input context, as this can leak information across comparisons. For position bias mitigation, always send each pair twice with reversed order and require consistent preference. If the two runs disagree, escalate the pair rather than picking one result.

Integration points depend on your evaluation architecture. In a batch RLHF data pipeline, the tie-breaker runs as a post-processing step after initial pairwise comparison. In a real-time champion-challenger system, it runs synchronously before returning a final preference to the metrics dashboard. In both cases, the harness should expose a tie_break(output_a, output_b, context, criteria_weights) function that returns a structured TieBreakResult with fields for preferred_output, justification, confidence, and escalation_required. Wire this function behind a feature flag so you can disable tie-breaking or adjust the invocation threshold without redeploying. Monitor the tie-breaker invocation rate—if it exceeds 20% of total comparisons, your primary judge likely needs recalibration rather than more tie-breaking.