Support Ticket Intent Disambiguation Prompt Template

The full unstructured text of the support ticket submitted by the customer, including subject line and description.

Subject: Cannot access admin panel after update Body: I updated to v4.2 this morning and now the admin panel shows a white screen. I've tried clearing cache but it didn't help. Also my reports aren't loading.

Non-empty string with minimum 10 characters. Reject tickets that are only whitespace, emoji, or attachments with no text. Log null or empty input as a pre-processing failure before model call.

A defined list of valid support categories the model can classify the ticket into. Acts as the output label space.

["Account Access", "Billing", "Bug Report", "Feature Request", "Integration", "Performance", "Security", "Onboarding"]

Must be a non-empty array of unique strings. Validate that no duplicate labels exist and that the list matches the current production taxonomy. Schema check: array of strings, min 2 items, max 50 items.

The support tier of the customer submitting the ticket, used to adjust response priority and tone.

"enterprise"

Must be one of a predefined enum: "free", "starter", "professional", "enterprise", "partner". Reject unknown values. Default to "starter" only if explicitly configured as a fallback; otherwise fail fast.

A condensed summary of the customer's last 3 support tickets, if available. Used to detect recurring issues or pattern escalation.

"2024-11-02: Admin panel slow loading (resolved via cache clear). 2024-12-15: Report export timeout (resolved via batch size reduction)."

Optional. If provided, must be a string under 500 tokens. If null, the prompt must not hallucinate prior history. Validate token length before assembly; truncate with a note if over budget.

The name of the support agent who will receive the disambiguation output, used to personalize the suggested clarification question.

"Jordan"

Non-empty string, max 50 characters. Strip leading/trailing whitespace. Reject strings containing only numbers or special characters. If null, use a generic placeholder like "the support agent" in the clarification question template.

The exact JSON schema the model must conform to in its response, including field names, types, required fields, and enum constraints.

{"type": "object", "properties": {"primary_intent": {"type": "string", "enum": ["Account Access", "Billing", "Bug Report", "Feature Request", "Integration", "Performance", "Security", "Onboarding"]}, "confidence": {"type": "number", "minimum": 0, "maximum": 1}, "alternative_intents": {"type": "array", "items": {"type": "object", "properties": {"intent": {"type": "string"}, "confidence": {"type": "number"}}, "required": ["intent", "confidence"], "maxItems": 3}}, "clarification_question": {"type": "string"}, "routing_decision": {"type": "string", "enum": ["auto_route", "clarify_first", "escalate"]}}, "required": ["primary_intent", "confidence", "alternative_intents", "clarification_question", "routing_decision"]}

Must be a valid JSON Schema object. Validate parseability before prompt assembly. Confirm that enum values in the schema match [TICKET_CATEGORIES]. Reject schemas missing required fields or with circular references.

The confidence score below which the model should recommend clarification over automatic routing.

0.75

Must be a float between 0.0 and 1.0. Validate range. If set to 0.0, the model will always recommend clarification; if 1.0, it will never recommend clarification. Log threshold value in trace for auditability.

The maximum character length of the generated clarification question to prevent verbose or multi-part questions.

150

Must be a positive integer between 50 and 300. Validate range. If the model output exceeds this length in post-processing, truncate and flag for review rather than silently passing through.

Array of objects

Array length >= 1 and <= 5. Each object must contain intent_label, confidence_score, and evidence_fragment.

String

Must match an entry in [ALLOWED_INTENT_CATEGORIES]. No fabricated labels allowed.

Number (0.0-1.0)

Must be a float between 0.0 and 1.0. Scores must be in descending order across the array.

String

Must be a verbatim or near-verbatim substring from [TICKET_BODY]. Null not allowed.

String

Must equal ranked_intents[0].intent_label. Used for quick routing checks.

Boolean

Must be true if top_intent confidence_score < [CONFIDENCE_THRESHOLD] or if second-highest score is within [MARGIN_THRESHOLD] of top score.

String

Required if ambiguity_flag is true. Must be a single interrogative sentence. Must not exceed 150 characters. Must not re-list all options.

String or null

If clarification_question is present, must identify which ranked_intents entry the question aims to disambiguate. Null if ambiguity_flag is false.

Top-ranked intent matches the ground-truth label from a held-out test set of 200+ tickets.

Top intent is incorrect, or the correct intent is ranked below position 3.

Run prompt against a golden dataset of manually labeled, ambiguous tickets. Measure Top-1 and Top-3 accuracy.

The generated question is specific enough that a non-technical customer can answer it with a single sentence.

Question is open-ended (e.g., 'Can you provide more details?'), asks for information already in the ticket, or requires technical knowledge.

Human review by a support team lead on a sample of 50 outputs. Score each question as 'Actionable' or 'Not Actionable'. Target >90% actionable.

The output is valid JSON matching the [OUTPUT_SCHEMA] exactly, with all required fields present and correctly typed.

JSON parsing fails, a required field like ranked_intents is missing, or a field contains an unexpected data type.

Automated schema validation in the test harness. Use a JSON Schema validator against the expected contract. Run on every CI build.

The confidence score for the top intent is >= 0.7 when the intent is unambiguous and < 0.7 when it is genuinely ambiguous.

High confidence (>= 0.9) is assigned to a wrong intent, or low confidence (< 0.5) is assigned to a clearly correct intent.

Compare the model's confidence score against human-rated ambiguity labels (1-5 scale). Calculate Expected Calibration Error (ECE).

A clarification question is generated only when the top intent's confidence is below the [CLARIFICATION_THRESHOLD].

A question is generated for high-confidence intents (annoying the agent), or no question is generated for a low-confidence, ambiguous ticket.

Assert that clarification_question is null when confidence > [CLARIFICATION_THRESHOLD] and is a non-empty string when confidence <= [CLARIFICATION_THRESHOLD].

All routing_tags in the output are present in the provided [VALID_ROUTING_TAGS] list.

A hallucinated or misspelled routing tag appears, which would break downstream automation.

Set-based comparison in the eval script. Check that the set of output tags is a subset of the allowed tags list.

Each intent in ranked_intents includes a rationale that quotes or directly references a phrase from the [TICKET_TEXT].

The rationale is generic ('The user seems frustrated'), hallucinates ticket content, or is missing entirely.

LLM-as-judge check: prompt a separate evaluator model to verify that each rationale is factually grounded in the provided ticket text.

For tickets with fewer than 10 words, the prompt abstains by setting the top intent to 'insufficient_information' and confidence to 0.0.

The model confidently classifies a near-empty ticket into a specific category, leading to a misroute.

Unit test with a set of 10 short, ambiguous tickets (e.g., 'It's broken', 'Help'). Assert the top intent is 'insufficient_information'.