Semantic Duplicate Clustering Prompt

This prompt is designed for research synthesis and feedback analysis teams who need to group semantically similar items from model-generated outputs into coherent clusters. It addresses the common post-generation problem where a flat list of items—such as user feedback comments, extracted research claims, or AI-generated ideas—contains near-duplicates that share meaning but use different wording. The prompt assigns a canonical representative per cluster, explains the grouping rationale, and provides cluster quality metrics, making it a critical repair step before human review or downstream analysis.

Use this prompt when you have an existing list of items and need to reduce redundancy without losing semantic diversity. It is not a generation-time constraint; it assumes the input list already exists and may contain items that are conceptually identical but lexically distinct. The prompt is ideal for analysts preparing data for thematic review, engineers cleaning model outputs before database insertion, and product teams consolidating user feedback into actionable themes. Do not use this prompt when you need exact string matching (use a hash-based deduplication script), when the input is a single long-form document requiring paragraph-level deduplication (use the Near-Duplicate Paragraph Detection Prompt), or when you need to merge structured JSON records by a primary key (use the Redundant Field Merge Prompt).

Before implementing, ensure you have a clear definition of the minimum cluster size and a strategy for handling singletons—items that don't cluster with anything else. The prompt works best when the input list contains 20–200 items; smaller lists may not benefit from clustering, and larger lists may hit context window limits. Pair this prompt with a validation harness that checks for cluster coherence and flags clusters with low internal similarity scores for human review. For high-stakes analysis where misclustering could distort research findings, always route low-confidence clusters to a human reviewer before finalizing the output.

This template is designed to be pasted directly into your system or user message. It instructs the model to group a list of input items into semantic clusters, select a single canonical representative for each cluster, and provide a clear rationale for every grouping decision. The prompt is parameterized with square-bracket placeholders that you must replace with your specific data, configuration, and constraints before execution. It is built for research synthesis and feedback analysis pipelines where raw model outputs contain redundant or thematically overlapping items that need to be collapsed into a structured, deduplicated set.

textCopy
You are a semantic clustering engine. Your task is to group the items provided in [INPUT_ITEMS] into clusters of semantically similar items. Two items belong in the same cluster if they express the same core idea, claim, finding, or feedback point, even if the wording differs.

## Input Items
[INPUT_ITEMS]

## Configuration
- Minimum Cluster Size: [MIN_CLUSTER_SIZE]
- Maximum Clusters: [MAX_CLUSTERS]
- Clustering Strictness: [STRICTNESS] (Options: 'high'—only group near-identical meanings; 'medium'—group paraphrases and same-topic items; 'low'—group broadly related items)
- Output Format: [OUTPUT_SCHEMA]

## Instructions
1. Analyze each item and identify its core semantic intent.
2. Group items into clusters based on the configured strictness level.
3. Discard any cluster smaller than the minimum cluster size. Leave those items unclustered.
4. For each valid cluster, select one canonical representative that best captures the shared meaning. Prefer the most specific and well-formed item.
5. Assign a concise, descriptive label to each cluster.
6. For each cluster, write a one-sentence rationale explaining why the items were grouped together.
7. If [INCLUDE_QUALITY_SCORES] is true, assign a cohesion score (1-10) to each cluster, where 10 means all items are near-identical in meaning.

## Constraints
- Do not force items into clusters if they do not meet the strictness threshold.
- An item may belong to only one cluster.
- If no valid clusters are found, return an empty clusters list and explain why.
- Output only the structure defined in the output schema. No additional commentary.

To adapt this template, replace each bracketed placeholder with concrete values. [INPUT_ITEMS] should be a JSON array of strings or objects with an id and text field. [MIN_CLUSTER_SIZE] is an integer, typically 2 or 3, that prevents singletons from being treated as clusters. [MAX_CLUSTERS] caps the total number of clusters returned, which is useful for downstream review queues. [STRICTNESS] directly controls the false-positive rate: use 'high' when you need near-exact matches only, and 'low' when you want to surface broad thematic groupings. [OUTPUT_SCHEMA] should be replaced with a concrete JSON schema or a reference to one, such as { "clusters": [{ "label": "string", "canonical_representative": "string", "item_ids": ["string"], "rationale": "string", "cohesion_score": number }], "unclustered_item_ids": ["string"] }. The [INCLUDE_QUALITY_SCORES] boolean lets you toggle the optional cohesion metric for eval pipelines. After pasting, validate that your input items are properly escaped for the model's tokenizer and that the output schema matches your downstream parser exactly. For high-stakes deduplication where a false merge would corrupt a dataset, always route the output through a human review step or an LLM judge with a calibrated rubric before ingestion.

The Semantic Duplicate Clustering Prompt is not a one-shot playground tool. In production, it operates as a post-generation repair step within a batch processing pipeline or an API endpoint that receives model outputs and returns clustered, deduplicated records. The prompt expects a list of items—typically strings, short paragraphs, or structured records—and produces clusters with a canonical representative, grouping rationale, and cluster quality metrics. Wiring this into an application requires a harness that validates input shape, enforces minimum cluster size controls, handles empty or single-item inputs gracefully, and gates ambiguous clusters for human review when confidence falls below a configured threshold.

The implementation harness should follow a validate → prompt → validate → route pattern. First, validate that the input is a non-empty array of items with a consistent schema. If the input contains fewer items than the configured [MIN_CLUSTER_SIZE], skip clustering and return the items as singletons. Second, construct the prompt by injecting the item list into the [INPUT_ITEMS] placeholder, setting [MIN_CLUSTER_SIZE] and [SIMILARITY_THRESHOLD] from application config, and including the [OUTPUT_SCHEMA] that defines the expected JSON structure: an array of cluster objects, each with a cluster_id, canonical_representative, member_indices, rationale, and confidence_score. Third, after receiving the model response, validate that every input item index appears exactly once across all clusters, that confidence scores are numeric and between 0 and 1, and that no cluster is smaller than the minimum size unless it contains singletons explicitly flagged as unclustered. Fourth, route clusters with confidence below [AUTO_RESOLVE_THRESHOLD] to a human review queue, while auto-accepting high-confidence clusters.

Retry logic must handle three failure modes. Schema violations: if the model returns malformed JSON or missing required fields, use a structured output repair prompt from the Output Repair and Validation pillar to recover the payload before retrying the clustering prompt once. Coverage gaps: if the union of all cluster member indices does not equal the set of input item indices, log the missing indices and retry with an explicit instruction to account for all items. Low-confidence flooding: if more than [MAX_REVIEW_QUEUE_RATIO] of clusters fall below the auto-resolve threshold, abort the batch and alert an operator—this signals either a threshold misconfiguration or input data that is too homogeneous for meaningful clustering. Log every clustering run with input hash, output cluster count, confidence distribution, and review queue depth for observability. Model choice matters: use a model with strong JSON adherence and semantic reasoning (such as Claude 3.5 Sonnet or GPT-4o) and set temperature=0 to minimize variance in cluster assignments.

Do not treat this prompt as a standalone deduplication engine for high-stakes use cases like financial transaction reconciliation or clinical record merging without human-in-the-loop approval on every cluster. The prompt identifies semantic similarity, not ontological identity—two items about the same topic are not necessarily the same fact. For regulated domains, configure [AUTO_RESOLVE_THRESHOLD] to 1.0, effectively routing all clusters to human review. For high-throughput pipelines, pair this prompt with the Exact Duplicate Removal Prompt as a pre-filter to strip verbatim duplicates before semantic clustering, reducing token usage and improving cluster quality. The next step after reading this harness is to implement the input validator and output schema checker as code-level guards before integrating the prompt call, then run a calibration batch against a labeled dataset to tune [SIMILARITY_THRESHOLD] for your domain.