Ambiguous Entity Resolution with Knowledge Base Prompt Template

This prompt is designed for the clarification layer of a RAG-augmented assistant, specifically to resolve ambiguous entity references from user input against a structured knowledge base. The core job-to-be-done is to prevent the system from hallucinating a single, incorrect match when a user's query—containing a name, code, or term—returns multiple high-similarity records from your retrieval pipeline. The ideal user is an AI engineer or product developer building a copilot, support agent, or data-retrieval assistant where entity linking precision is critical for downstream task success. You should use this prompt only after your retrieval step has returned multiple candidates above a defined similarity threshold, and you have their associated metadata (e.g., IDs, descriptions, distinguishing attributes) ready to inject into the prompt's context.

The prompt's operational boundary is strict: it assumes you have already retrieved candidate entities with scores and metadata, and its sole function is to format them into a disambiguation question. Do not use this prompt when only one candidate exists above your threshold, or when the user's reference is unambiguous—in those cases, proceed directly to answer generation. In regulated domains like healthcare or finance, you must log the disambiguation decision, the evidence that supported each candidate, and the user's selection to maintain a complete audit trail. A common failure mode is using this prompt without a similarity threshold, which floods the context window with irrelevant options. Always pre-filter candidates with a minimum score (e.g., >0.75) to keep the prompt's input tight and the user's cognitive load low.

Before wiring this into your application, define your evaluation criteria for success: the model should never pick a winner on its own, and the disambiguation question must highlight the distinguishing attribute between the top candidates, not just re-list all metadata. If the model consistently fails to generate a useful distinguishing question, the problem is often upstream in your retrieval metadata quality, not in the prompt itself. Proceed to the prompt template section to copy the exact instructions, and then to the implementation harness to learn how to validate the output, handle retries, and log decisions before the user's selection is passed to the answer-generation layer.

This prompt template is designed for RAG-augmented assistants that must resolve ambiguous entity references from user input against a structured knowledge base. It takes a user query containing an ambiguous entity mention, a set of candidate entities retrieved from your knowledge base, and a similarity threshold. The prompt instructs the model to either link to the best match when confidence is high or generate a precise disambiguation question when multiple candidates exceed the threshold. The output is a structured JSON object containing the resolution decision, candidate matches, and, if necessary, a clarification question that highlights the distinguishing feature between the top candidates.

textCopy
You are an entity resolution assistant for a knowledge base. Your job is to resolve ambiguous entity references in user queries.

## INPUT
User Query: [USER_QUERY]
Candidate Entities (from knowledge base): [CANDIDATE_ENTITIES_JSON]
Similarity Threshold: [SIMILARITY_THRESHOLD]

## TASK
1. Compare the ambiguous entity mention in the user query against each candidate entity.
2. If exactly one candidate has a similarity score above the threshold, link to that entity.
3. If multiple candidates exceed the threshold, generate a targeted disambiguation question that highlights the key distinguishing feature between the top candidates.
4. If no candidate exceeds the threshold, indicate that no match was found.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "resolution_type": "linked" | "clarification_needed" | "no_match",
  "linked_entity": {
    "id": "string | null",
    "name": "string | null",
    "confidence": "number | null"
  },
  "candidates": [
    {
      "id": "string",
      "name": "string",
      "similarity_score": "number",
      "distinguishing_feature": "string"
    }
  ],
  "clarification_question": "string | null",
  "reasoning": "string"
}

## CONSTRAINTS
- Never fabricate entity IDs or names not present in the candidate list.
- The clarification question must be a single sentence that helps the user distinguish between the top candidates without repeating all their attributes.
- If resolution_type is "linked", the clarification_question field must be null.
- If resolution_type is "clarification_needed", the linked_entity fields must be null.
- Base your decision on the similarity scores provided; do not re-score candidates.

Adapt this template by replacing the square-bracket placeholders with data from your retrieval pipeline. [USER_QUERY] should contain the raw user message. [CANDIDATE_ENTITIES_JSON] must be a pre-serialized JSON array of candidate objects, each with at least an id, name, and similarity_score field produced by your retrieval system. [SIMILARITY_THRESHOLD] is a float between 0 and 1 that you tune based on your eval results. If your knowledge base includes additional metadata such as entity type, department, or location, add those fields to the candidate objects and update the output schema accordingly. For high-stakes domains like healthcare or finance, add a [RISK_LEVEL] placeholder and include instructions to escalate to human review when the top candidate scores fall within a marginal band below the threshold. Always validate the model's output against the schema before surfacing the clarification question to the user; a malformed JSON response should trigger a retry or fallback to a generic disambiguation prompt.

This prompt is designed to sit inside a retrieval-augmented generation (RAG) pipeline, not as a standalone chat interaction. The application layer is responsible for executing the initial knowledge base search, passing the top-k candidate entities and their metadata into the prompt's [CANDIDATE_ENTITIES] placeholder, and then parsing the structured output. The prompt itself does not perform the search; it reasons over the results you provide. This separation is critical because it allows you to control retrieval quality, latency, and cost independently from the LLM's disambiguation logic.

The implementation harness should enforce a strict contract around the prompt's output. Expect a JSON object with a decision field set to either link, clarify, or no_match. If link, the harness must validate that the linked_entity_id exists in the provided candidates. If clarify, the harness must confirm that the clarification_question is non-empty and that the candidate_options array contains at least two distinct IDs from the input. A post-processing validation layer should reject any response that violates this schema and trigger a single retry with the validation error injected into the [CONSTRAINTS] block. After one failed retry, the system should escalate to a no_match state or a human review queue rather than looping indefinitely.

For high-stakes domains like healthcare or finance, the clarify path should never autonomously execute a destructive action. Instead, surface the clarification question to the user in the UI and pause the workflow. Log the full prompt, the retrieved candidates, the model's raw output, and the validation result for every disambiguation event. This audit trail is essential for debugging retrieval gaps, identifying when the similarity threshold needs tuning, and proving to reviewers that the system did not silently guess an entity. When deploying, start with a high similarity threshold for candidate inclusion and monitor the ratio of clarify to link decisions to find the right balance between precision and user friction.