Review Item Deduplication Prompt for Queue Cleanup

This prompt is designed for the specific job of cleaning up a backlogged or continuously high-volume human review queue where the same underlying issue surfaces multiple times from different automated sources, monitoring systems, or user reports. The core job-to-be-done is to reduce reviewer fatigue and accelerate mean time to resolution by collapsing near-duplicate items into a single, canonical review task. The ideal user is an operations lead or platform engineer who owns the queue's throughput and quality, and who can integrate this batch reasoning step into an existing triage pipeline. You need a batch of items, each with enough text content—such as a title, description, and any associated error logs or user messages—for the model to perform a semantic comparison. This is not a real-time, single-item deduplication tool; that problem is better solved with vector search and indexing against a database of known issues.

Before using this prompt, ensure your input data is properly staged. Each item in your batch should be a structured object with a unique identifier and a text field containing the core signal for comparison. The prompt works best on batches of 20 to 100 items, where the model can reason holistically about the entire set. For larger queues, you should implement a pre-fetch step that retrieves a candidate pool of potentially similar items, perhaps using a lightweight keyword or embedding search, and then use this prompt to perform the final, high-precision clustering on that subset. The output is a set of cluster assignments, a similarity rationale for each cluster, and a selected canonical item that represents the cluster for review. This output is designed to be consumed by your application code, which should then suppress or link the duplicate items in the review queue and present only the canonical item to a human reviewer.

Do not use this prompt when the cost of a false merge—incorrectly grouping two distinct issues as one—is extremely high and cannot be mitigated by a quick human sanity check. In such high-stakes scenarios, the prompt's output should be treated as a suggestion, not an automated action. A safer pattern is to present the proposed clusters to a reviewer for confirmation before merging, turning the prompt into a 'deduplication suggestion' tool. Also, avoid this approach for items where the differentiating details are buried in long, unstructured attachments like PDFs or images; the prompt's effectiveness is bounded by the quality and relevance of the text you provide in the item's content field. For those cases, invest in a pre-processing step to extract and summarize the key details before running deduplication.

The next step after reading this introduction is to examine the prompt template itself. You'll see how to structure the input batch, define the output schema for your clusters, and set constraints like a minimum similarity threshold. From there, you can move to the implementation harness section to learn how to wrap this prompt in validation logic, handle edge cases like items that don't fit any cluster, and integrate the results safely into your production review queue.

This prompt template is designed to be dropped into your deduplication workflow. It instructs the model to analyze a batch of review items, identify clusters of duplicates or near-duplicates, articulate the reasoning behind each grouping, and nominate a single canonical item that best represents the cluster. The output is structured for direct ingestion by a review queue system or a human triage dashboard. Before using this prompt, ensure your input data is clean and that each item has a stable unique identifier; the model relies on these IDs to reference items in its output.

textCopy
You are an expert deduplication analyst preparing a review queue for human triage. Your task is to analyze a batch of review items and identify clusters of duplicates or near-duplicates. For each cluster, explain the similarity reasoning and select one canonical item that best represents the group for human review.

## INPUT
A JSON array of review items. Each item has an `id` (string), `title` (string), `description` (string), and `source` (string).

[INPUT]

## CONSTRAINTS
- Group items only when they describe the same underlying issue, request, or event. Superficial keyword overlap is not enough.
- An item can belong to only one cluster.
- If an item is unique, place it in a cluster of size 1.
- For each cluster, select the canonical item based on: (1) most complete description, (2) clearest articulation of the core issue, and (3) presence of actionable details.
- Do not merge items that are distinct but related (e.g., a bug report and a feature request about the same component).

## OUTPUT_SCHEMA
Return a valid JSON object with a single key "clusters" containing an array of cluster objects. Each cluster object must have:
- "cluster_id" (string): a unique identifier for the cluster, e.g., "cluster-1".
- "canonical_item_id" (string): the `id` of the selected canonical item.
- "member_ids" (array of strings): all `id`s in the cluster, including the canonical item.
- "similarity_reasoning" (string): a concise explanation of why these items were grouped, referencing specific shared details.
- "distinctness_check" (string): a brief note confirming why this cluster is distinct from other clusters in the batch.

## RISK_LEVEL
High. False merges can hide distinct issues from reviewers. Missed duplicates waste reviewer time. When uncertain, default to keeping items separate and flag the ambiguity in `similarity_reasoning`.

## EXAMPLES
Input:
[
  {"id": "item-1", "title": "Login button not working", "description": "The login button on the main page does nothing when clicked. Chrome 120.", "source": "support"},
  {"id": "item-2", "title": "Cannot log in", "description": "Clicking login does not respond. Using latest Chrome.", "source": "twitter"},
  {"id": "item-3", "title": "Dark mode request", "description": "Please add a dark mode option to settings.", "source": "feedback"}
]

Output:
{
  "clusters": [
    {
      "cluster_id": "cluster-1",
      "canonical_item_id": "item-1",
      "member_ids": ["item-1", "item-2"],
      "similarity_reasoning": "Both items describe the login button being unresponsive on Chrome. Item-1 provides the specific browser version and page location, making it the more complete report.",
      "distinctness_check": "This cluster is about a login button bug. Distinct from cluster-2, which is a feature request."
    },
    {
      "cluster_id": "cluster-2",
      "canonical_item_id": "item-3",
      "member_ids": ["item-3"],
      "similarity_reasoning": "Single item requesting a dark mode feature. No duplicates found.",
      "distinctness_check": "This is a feature request, distinct from the bug report in cluster-1."
    }
  ]
}

To adapt this prompt, replace the [INPUT] placeholder with your serialized JSON array of review items. If your items have different fields, update the field descriptions in the INPUT section and adjust the canonical selection criteria in CONSTRAINTS to reference your schema. The OUTPUT_SCHEMA is strict JSON; validate the response against this schema before forwarding clusters to your review queue. If your deduplication logic requires domain-specific rules—such as matching by customer ID or time window—add those as explicit constraints. Always run eval checks for false merges (distinct items incorrectly grouped) and missed duplicates (true duplicates left in separate clusters) before deploying changes to this prompt.

This prompt is designed to run as a batch or streaming processor over a set of review items, not as a single-turn chat interaction. The implementation harness should treat each call as an idempotent clustering operation: given a list of items, return cluster assignments. The harness is responsible for chunking large queues into manageable batches, tracking which items have been processed, and merging results across batches without creating cross-batch duplicate fragmentation.

Integration pattern: Build a service function that accepts a list of review item objects (each with an id, title, description, and any relevant metadata) and returns a list of cluster assignments. The function should: (1) validate that each input item has the required fields before calling the model; (2) call the prompt with a batch size that fits within the model's context window—typically 50–100 items per call for most models; (3) parse the JSON output and validate that every input item ID appears exactly once in the output clusters; (4) log any items that were not assigned or were assigned to multiple clusters as validation failures. For high-volume queues, implement a sliding window approach where each batch overlaps with the previous batch by 10–20% to catch cross-batch duplicates, then merge overlapping clusters using a union-find or connected-components algorithm in the application layer.

Model choice and retries: Use a model with strong JSON mode and reasoning capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Set temperature to 0 or a very low value (0.1) to maximize deterministic clustering. If the output fails JSON schema validation, retry once with the validation error message appended to the prompt as feedback. If the second attempt also fails, log the batch for human review rather than silently dropping items. For cost-sensitive deployments, consider a two-stage pipeline: first use a cheaper embedding model to compute cosine similarity between item pairs, then use this prompt only on the high-similarity candidate pairs to confirm or reject duplicate status with reasoning.

Human review gates: The prompt output includes a canonical_item for each cluster and a similarity_reasoning field. Before automatically merging or closing items based on the output, route clusters where the similarity reasoning contains hedging language (e.g., 'possibly', 'might be', 'unclear if') to a human reviewer. Implement a confidence threshold: if the model's own reasoning suggests ambiguity, do not auto-merge. Additionally, maintain an audit log of all merge actions with the cluster ID, the canonical item selected, the items merged, and the similarity reasoning, so that operations leads can spot-check decisions and reverse incorrect merges.

What to avoid: Do not use this prompt on items that require domain-specific legal, medical, or financial judgment to determine if they are duplicates—surface similarity may mask materially distinct cases. Do not auto-close items without preserving the original item IDs and a link to the merge decision. Finally, avoid running this prompt on queues where items are still actively being updated, as new information may change duplicate status after clustering is complete.