Exact Duplicate Removal Prompt Template

This prompt is a targeted repair tool for a specific failure mode: verbatim string duplication in model outputs. It is designed for data pipeline engineers and platform teams who have already extracted a delimited list from a model response and discovered identical entries. The core job-to-be-done is cleaning a batch of strings by removing exact duplicates while preserving the original order and providing count metadata for downstream validation. This is a post-generation repair step, not a generation-time constraint. You should reach for this prompt when your pipeline has produced a list of items—such as generated tags, extracted entities, or synthesized bullet points—and a quality check reveals repeated strings that will corrupt deduplication logic, inflate counts, or pollute a downstream database with redundant records.

The ideal user has a string input that is already formatted as a delimited list (newline, comma, or custom delimiter) and needs a clean, order-preserved result. The prompt assumes the input is a single string, not a structured object. It does not handle fuzzy matching, semantic deduplication, or cross-record entity resolution. If you need to merge near-duplicate paragraphs, consolidate entities with multiple surface forms, or deduplicate JSON objects by identity fields, use the sibling playbooks for those tasks. This prompt also does not validate whether the input itself is well-formed; if the model output is truncated, contains encoding artifacts, or mixes delimiters, you should apply the relevant repair playbooks from the Output Repair and Validation pillar before invoking this deduplication step.

Before using this prompt, ensure you have a clear definition of what constitutes a duplicate in your context. The prompt performs exact string matching, which means trailing whitespace, case differences, or punctuation variations will cause two semantically identical entries to be treated as distinct. If your pipeline requires case-insensitive or whitespace-normalized matching, preprocess the input before passing it to this prompt, or pair it with the Canonicalization and Deduplication Combo Prompt. After deduplication, always validate the output by comparing the pre- and post-deduplication counts provided in the response metadata. If the deduplication ratio is unexpectedly high or low, investigate the input formatting before assuming the prompt failed. For high-risk pipelines where data integrity is critical, log the before and after states and consider adding a human review step for batches where the deduplication rate exceeds a configurable threshold.

This template is designed for data pipeline engineers who need a reliable, stateless repair step for batch model outputs. It targets the specific failure mode where a model returns a list, array, or newline-delimited set of records containing identical string entries. The prompt instructs the model to act as a deterministic deduplication function, removing subsequent occurrences of any string while keeping the first occurrence in its original position. It also requires the model to return pre- and post-deduplication counts, which serves as a critical validation signal for your application harness.

textCopy
You are an exact deduplication function. Your task is to process a list of records and remove any duplicate string entries.

INPUT:
[INPUT_LIST]

INSTRUCTIONS:
1. Iterate through the input list in order.
2. Keep the first occurrence of every unique string.
3. Remove all subsequent identical occurrences of that string.
4. Preserve the original relative order of the kept items.
5. Do not alter the content, casing, or whitespace of any kept string.
6. Treat an empty string as a valid unique value.

OUTPUT_SCHEMA:
Return a valid JSON object with the following structure:
{
  "original_count": <integer>,
  "deduplicated_count": <integer>,
  "removed_count": <integer>,
  "deduplicated_list": [<string>, ...]
}

CONSTRAINTS:
- If the input list is empty, return an empty list with all counts set to 0.
- If no duplicates are found, the deduplicated_list must be identical to the input and removed_count must be 0.
- Do not include any explanatory text outside the JSON object.

To adapt this template, replace [INPUT_LIST] with your actual data. The input should be a valid JSON array of strings. If your source data is a newline-delimited text block, pre-process it into a JSON array before calling the model. For high-throughput pipelines, consider adding a pre-check in your application code: if the list length is below a configurable threshold, perform deduplication in code without calling the model to save latency and cost. After receiving the model's JSON response, always validate that original_count matches the length of the input you sent and that original_count - removed_count == deduplicated_count. A mismatch indicates a model error and should trigger a retry or fallback to a deterministic code path.

The exact duplicate removal prompt is designed to be a stateless, idempotent repair step in a batch processing pipeline. It accepts a list of strings and returns a deduplicated list preserving first-occurrence order. This makes it straightforward to integrate as a post-generation filter: model output flows into the prompt, deduplicated output flows to the next stage. The implementation harness must handle three concerns beyond the prompt itself: input validation (ensuring the input is a valid list), output validation (confirming the deduplicated list is a subset with no lost unique entries), and observability (logging deduplication counts for pipeline monitoring).

Wire the prompt as a synchronous repair call with a strict timeout. In pseudocode: deduped = await deduplicate(input_list, timeout=10s). Before calling the prompt, validate that the input is a non-null array of strings. If the input is empty, short-circuit and return an empty list without calling the model—this avoids unnecessary latency and cost. After receiving the model response, run a post-deduplication validator that checks: (1) the output is a valid array of strings, (2) every element in the output exists in the input, (3) no element appears more than once in the output, and (4) the set of unique values in the output matches the set of unique values in the input. If validation fails, retry once with the same input and a stronger constraint instruction appended. If the retry also fails, log the failure with the input hash and fall back to a deterministic programmatic deduplication using a language-native Set or equivalent. The model-based approach is the primary path; the deterministic fallback is the safety net.

For observability, emit a structured log record after each deduplication operation containing: input_count, output_count, duplicates_removed, processing_time_ms, model_used, retry_attempts, and fallback_used (boolean). This lets you track deduplication rates over time and detect model drift—if the fallback rate suddenly increases, the model may be degrading. For high-throughput pipelines, consider batching multiple deduplication requests into a single model call with a delimiter-separated format to reduce API overhead, but keep individual batch sizes small enough that a single failure doesn't block a large portion of the workload. Always use idempotency keys derived from the input hash so that retries don't produce duplicate side effects in downstream systems.