Duplicate-Free List Generation Constraint Prompt

This playbook is for prompt engineers and platform developers who need to prevent duplicate entries at generation time rather than repairing them after the fact. Use this constraint prompt pattern when your application requires the model to produce lists, sets, or enumerated outputs where repeated items would corrupt downstream processing, inflate counts, or degrade user experience. The prompt instructs the model to self-check for duplicates before emitting output and includes a self-verification step that makes the constraint observable. This approach works best when the list cardinality is moderate (under 50 items) and the model can hold the full output in working memory for the self-check pass. For batch pipelines generating thousands of records, pair this with a post-generation deduplication harness instead of relying solely on generation-time constraints.

The ideal user for this pattern is an integration engineer or backend developer building a feature that surfaces AI-generated lists directly to users or into a strict database schema. Required context includes a clear definition of what constitutes a duplicate for your domain—is it an exact string match, a case-insensitive match, or a semantic equivalence? You must also define the expected cardinality and the output format. This prompt is not a replacement for database UNIQUE constraints or application-level idempotency checks; it is a generation-time quality gate that reduces but does not eliminate the need for defensive downstream validation. When the cost of a duplicate reaching the user is high, always add a post-generation uniqueness assertion in your application harness.

Do not use this prompt pattern when the list is unbounded, when duplicates are acceptable with different metadata, or when the model cannot feasibly hold the entire output in context for the self-verification pass. For streaming responses, the self-check step is impossible without buffering the full output, which defeats the purpose of streaming. In those cases, use a post-processing deduplication module. Similarly, avoid this pattern for lists where 'duplicate' requires external knowledge—such as determining that 'Apple Inc.' and 'AAPL' refer to the same entity—since the model's internal knowledge may be inconsistent. For those scenarios, pair the prompt with a retrieval step that provides a canonical entity list before generation begins.

This template enforces duplicate-free list generation at the model level, shifting the burden from post-processing repair to in-generation constraint adherence. The prompt uses a two-phase structure: first, the model generates the requested list under strict uniqueness rules; second, it performs a mandatory self-verification pass to confirm no duplicates exist before emitting the final output. This pattern is most effective when the list domain is well-defined, the uniqueness criteria are unambiguous, and the expected list length is small enough for the model to hold in working memory during verification.

textCopy
SYSTEM: You are a precise list generator. Your output must contain zero duplicate entries. Before responding, you must self-verify that every entry is unique according to the specified criteria.

USER: Generate a list of [LIST_DESCRIPTION].

CONSTRAINTS:
- Each entry must be unique based on [UNIQUENESS_CRITERIA].
- Do not include entries that are semantically equivalent even if phrased differently.
- Maximum [MAX_ENTRIES] entries.
- Output format: [OUTPUT_FORMAT].

SELF-VERIFICATION STEP (perform internally before responding):
1. Review every generated entry.
2. Check each pair of entries against the uniqueness criteria.
3. If any duplicates are found, remove them and keep only the first occurrence.
4. If removal reduces the list below [MIN_ENTRIES], generate replacement entries that satisfy the constraints.
5. Re-verify the final list.

After verification, output ONLY the final verified list with no additional commentary.

[ADDITIONAL_CONTEXT]

Adapt this template by tightening the uniqueness criteria for domains where near-duplicates are common. For example, when generating product feature lists, specify that entries differing only by synonyms or minor rephrasing count as duplicates. When the list domain is open-ended, add a negative constraint listing known duplicate patterns to avoid. For production reliability, pair this prompt with a post-generation validation harness that programmatically checks for exact and fuzzy duplicates, logs violation rates, and triggers retries with stricter constraints when the self-verification step fails. Avoid relying solely on self-verification for lists exceeding roughly 20 items, as model attention to fine-grained uniqueness degrades with length.

This prompt is designed to be used as a pre-generation constraint, not a post-hoc repair tool. The ideal integration point is inside a function that constructs the full model request, where you inject the constraint instructions and the self-verification step directly into the system or user message before calling the model. You should treat the prompt as a wrapper around the user's actual list-generation task: the user provides the topic and count, and your application layer inserts those values into the [LIST_TOPIC] and [DESIRED_COUNT] placeholders. The model is then instructed to generate, self-check, and emit only the final verified list.

A production harness requires three layers of validation beyond the model's self-check. First, a structural validator must confirm the output is a valid JSON array of strings with the expected length. Second, a semantic deduplication check should normalize entries (lowercase, trim, strip punctuation) and confirm that no two normalized forms are identical. Third, for high-reliability systems, run a similarity threshold check using embeddings or a cheap string metric (e.g., Levenshtein distance) to catch near-duplicates the model might have missed. If any validation layer fails, the harness should trigger a retry loop. A recommended pattern is to append the validation error message to the next request as additional context: "Previous output failed duplicate check. Found repeated entries: [X, Y]. Please regenerate with strict uniqueness." Limit retries to 3 attempts before escalating to a fallback model or a human review queue.

For observability, log every generation attempt with the input parameters, the raw model output, the validation results, and the retry count. This data is essential for tuning the prompt over time. Track two key metrics: constraint adherence rate (percentage of first attempts that pass all duplicate checks) and retry success rate (percentage of failed attempts that succeed on retry). If the constraint adherence rate drops below 90%, the prompt wording likely needs adjustment or the model is struggling with the specific list domain. For agent workflows, expose the deduplication result as a structured tool return so downstream agents can decide whether to use the list, request regeneration, or merge with other sources. Avoid using this prompt for lists longer than 50 items in a single call; batch the work or use a programmatic deduplication post-processor for large-scale list generation.