Ambiguity Handling Assessment in Reasoning Prompt

This prompt is built for evaluation engineers and AI safety teams who need to score how a reasoning trace manages ambiguous inputs. The job-to-be-done is automated quality assurance: you have a model that produces chain-of-thought reasoning, and you need to verify that it doesn't silently resolve ambiguity in a single direction without acknowledgment. Use this when the input to the reasoning model is inherently underspecified—think vague user requests, conflicting constraints, missing parameters, or terms with multiple valid interpretations. The prompt produces structured scores for ambiguity recognition, clarification-seeking behavior, assumption documentation, and multi-interpretation handling quality. It is not a general reasoning quality grader; if your inputs are unambiguous and the task is purely deductive, use the Step-by-Step Correctness Grading Prompt instead.

The ideal user is someone integrating this into an automated evaluation harness—typically an AI engineer or evaluation lead running a judge model against a dataset of reasoning traces paired with their original ambiguous inputs. You will need three things wired into the prompt: the original ambiguous [INPUT] that was given to the reasoning model, the full [REASONING_TRACE] produced by that model, and a [RUBRIC] defining your scoring dimensions and anchors. The prompt works best with a capable judge model (GPT-4, Claude 3.5, or equivalent) because it requires meta-reasoning about another model's reasoning. Do not use this prompt on traces shorter than a few sentences; it needs enough material to detect whether ambiguity was acknowledged. For high-stakes domains like healthcare or legal analysis, always route low scores to human review rather than treating the judge's output as a final gate.

Before deploying this prompt, define what 'good enough' means for your use case. A reasoning trace that scores 4/5 on ambiguity recognition but fails to document its assumptions might be acceptable for a low-risk summarization feature but unacceptable for a clinical decision support tool. Set explicit pass/fail thresholds in your evaluation harness, log every score with the trace ID and judge model version, and periodically calibrate the judge against human ratings using the Score Calibration and Judge Alignment Prompts playbook. The most common failure mode is the judge model hallucinating ambiguity where none exists—always include a few unambiguous control cases in your test suite to detect this drift.

This prompt template is designed to be copied directly into your evaluation harness. It instructs an LLM judge to assess a reasoning trace against a structured rubric for ambiguity handling. The template uses square-bracket placeholders for all dynamic inputs, making it safe to inject variables from your application code without confusing the model's instruction hierarchy.

textCopy
You are an expert evaluator grading the quality of ambiguity handling in a reasoning trace. Your task is to assess how well the trace recognizes, documents, and manages ambiguous or underspecified inputs.

## INPUT

[REASONING_TRACE]

## CONTEXT

[ORIGINAL_QUERY]

[AVAILABLE_EVIDENCE]

## EVALUATION RUBRIC

[RUBRIC]

## OUTPUT SCHEMA

Return a single JSON object with the following structure:
{
  "ambiguity_recognition_score": <integer 1-5>,
  "clarification_seeking_score": <integer 1-5>,
  "assumption_documentation_score": <integer 1-5>,
  "multi_interpretation_handling_score": <integer 1-5>,
  "overall_score": <integer 1-5>,
  "identified_ambiguities": [
    {
      "ambiguity_description": "<string>",
      "step_index": <integer or null>,
      "handling_quality": "<recognized_and_addressed | recognized_but_ignored | unrecognized>"
    }
  ],
  "critical_failures": ["<string>"],
  "justification": "<string>"
}

## CONSTRAINTS

[CONSTRAINTS]

## INSTRUCTIONS

1. Read the original query and available evidence to understand the ambiguity landscape.
2. Examine the reasoning trace step by step.
3. For each ambiguity dimension, assign a score using the rubric anchors.
4. List every ambiguity you identify, noting whether the trace recognized and addressed it, recognized but ignored it, or failed to recognize it entirely.
5. Flag any critical failures where the trace made a decision based on an unstated assumption that materially affects the conclusion.
6. Provide a concise justification for your overall score.

To adapt this template, replace each placeholder with your application's data. The [RUBRIC] placeholder should contain your scoring anchors for each dimension—for example, defining what a score of 1 versus 5 means for ambiguity recognition. The [CONSTRAINTS] placeholder can include domain-specific rules, such as requiring human review when critical failures are detected or when the overall score falls below a threshold. If your evaluation pipeline uses a specific JSON schema validator, ensure the output schema matches exactly. For high-stakes domains like healthcare or legal review, add a constraint that any trace with a critical failure must be escalated for human review before the score is accepted.

The Ambiguity Handling Assessment prompt is designed to operate as a model-graded evaluation step inside a larger reasoning trace QA pipeline. It expects a reasoning trace and the original ambiguous input as arguments, and it returns a structured JSON scorecard. This prompt is not a real-time user-facing prompt; it is an offline or async evaluation prompt that runs after a reasoning model has produced its output. Wire it into your evaluation harness as a post-generation check, not as a pre-response guardrail. The prompt works best with models that have strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable defaults. Avoid small or fast models that may struggle with the multi-dimensional rubric and produce inconsistent JSON.

Integration pattern: Place this prompt inside an evaluation loop that iterates over a dataset of reasoning traces paired with their original ambiguous inputs. For each pair, call the LLM Judge with the prompt template, passing [REASONING_TRACE] and [AMBIGUOUS_INPUT] as the primary variables. The [EVALUATION_RUBRIC] variable should be populated from your organization's ambiguity handling standards—define explicit scoring anchors for each dimension (Ambiguity Recognition, Clarification Seeking, Assumption Documentation, Multi-Interpretation Handling) on a 1–5 scale. The [OUTPUT_SCHEMA] variable must contain a strict JSON schema that the judge must follow, including required fields for overall_score, dimension_scores, flags, and justification. Validation layer: After receiving the judge's output, validate the JSON structure programmatically. Check that all required fields are present, scores fall within the defined range, and flags are from an allowed enum. If validation fails, retry up to two times with the same prompt, appending the validation error message to the [CONSTRAINTS] variable. After three failures, route the trace to a human review queue with the raw judge output and validation errors attached.

Logging and observability: Log every evaluation call with the trace ID, judge model version, prompt version, raw response, validation status, and final scores. This audit trail is critical because ambiguity handling assessments are subjective—two judges may disagree, and you need the evidence to calibrate your rubric over time. Store these logs in your evaluation database alongside the reasoning trace itself. Human review gates: For high-stakes domains (healthcare, legal, finance), configure a threshold rule: if the overall_score falls below 3 or any flags array contains CRITICAL_AMBIGUITY_MISSED, automatically escalate to human review regardless of validation success. This ensures that borderline or dangerous ambiguity misses don't slip through automated gates. Model choice trade-offs: If cost is a concern, use a smaller judge model (GPT-4o-mini, Claude Haiku) for initial scoring and escalate to a larger judge model only when scores are borderline or flags are raised. This two-tier approach reduces evaluation costs while maintaining quality on difficult cases. What to avoid: Do not use this prompt as a real-time guardrail that blocks model responses—ambiguity assessment is too nuanced for sub-second decisions. Do not skip the validation layer; LLM judges produce malformed JSON more often than you expect. Do not treat the judge's scores as ground truth without periodic human calibration samples.