Duplicate Sentence Stripping Prompt Template

This prompt is designed for text generation pipeline operators who need to clean long-form model outputs that contain verbatim repeated sentences. It is a post-generation repair tool, not a generation-time constraint. The primary job-to-be-done is to sanitize a single text block—such as a summarization result, an article draft, or a batch-generated report—by removing exact duplicate sentences before the content reaches a user, database, or downstream system. The prompt preserves the first occurrence of each sentence and maintains the original sequential flow, making it safe for narrative text where sentence order is critical for coherence.

Use this prompt when your model exhibits looping, echoing, or redundant restatement behaviors within a single response. For example, a summarization model might repeat a key finding verbatim in three consecutive paragraphs, or a long-form generator might get stuck in a degenerative loop. This prompt is ideal for insertion into a validation and repair step in an inference pipeline: after the model generates a response, a lightweight script can detect repeated sentences and, if found, invoke this prompt to clean the output. You should wire this into an automated harness that checks for duplication before calling the repair prompt to avoid unnecessary latency and cost. The prompt template accepts a [TEXT_INPUT] containing the raw model output and an optional [EXCEPTIONS] list for sentences that should be preserved despite repetition.

Do not use this prompt for removing intentional repetition such as poetic refrains, marketing taglines, or emphasis through anaphora unless you explicitly provide those sentences in the [EXCEPTIONS] placeholder. It is also unsuitable for deduplicating structured data records, JSON arrays, or lists of entities—for those tasks, use a dedicated record deduplication prompt from this pillar. Avoid using this as a streaming fix; it requires the full text block to function correctly. If your pipeline produces streaming output, buffer the complete response before applying this repair. Finally, for high-stakes content where information loss could have regulatory or safety implications, always pair this automated repair with a human review step or a post-repair evaluation that checks for missing key claims against the source material.

The following prompt template is designed to be dropped directly into your application's prompt layer. It instructs the model to scan the provided text, identify sentences that are repeated verbatim, and return the cleaned text with only the first occurrence of each sentence retained. The template uses square-bracket placeholders for all variable inputs, making it straightforward to wire into a templating engine or configuration file.

textCopy
You are a precise text editor. Your task is to remove verbatim duplicate sentences from the provided text.

## INPUT TEXT
[INPUT_TEXT]

## INSTRUCTIONS
1. Read the entire input text carefully.
2. Identify any sentence that appears more than once verbatim (character-for-character identical).
3. Keep only the **first occurrence** of each duplicated sentence in its original position.
4. Remove all subsequent verbatim duplicates completely.
5. Preserve all non-duplicate sentences, paragraph breaks, and the original logical flow.
6. Do not alter, paraphrase, or rewrite any sentence. Only remove exact duplicates.
7. If no duplicates exist, return the original text unchanged.

## IMPORTANT DISTINCTIONS
- **Intentional repetition** such as refrains, emphasis, or rhetorical devices that use identical wording should be treated as duplicates and removed unless explicitly listed in [PRESERVE_PATTERNS].
- Sentences that are semantically similar but not character-for-character identical are **not** duplicates. Do not remove them.
- Whitespace differences (trailing spaces, tabs) do not make sentences different. Normalize whitespace before comparison.

## PRESERVE PATTERNS
[PRESERVE_PATTERNS]

## OUTPUT FORMAT
Return only the cleaned text with duplicates removed. Do not add commentary, explanations, or markup.

## STREAMING NOTE
If processing in chunks, maintain a running set of seen sentences across chunks to catch duplicates that span chunk boundaries.

Adapting the template: Replace [INPUT_TEXT] with the full text output from your model. The [PRESERVE_PATTERNS] placeholder accepts a list of sentence patterns that should be kept even if repeated—for example, a recurring product tagline, a legal disclaimer that must appear in each section, or a poetic refrain. If no patterns need preservation, replace this placeholder with the word 'None.' For streaming pipelines, implement a sentence-level deduplication set outside the prompt and pass the accumulated seen-sentences as additional context if the model needs awareness of prior chunks. Always validate the output by counting sentences before and after to confirm the expected reduction.

The duplicate sentence stripping prompt is designed to sit in a post-generation repair loop, not as a standalone interactive tool. After your primary model produces long-form text—such as a summarization, report, or narrative generation—you route the raw output through this prompt before it reaches the user or downstream storage. The harness should treat the stripping step as a validation gate: if the prompt succeeds, the cleaned text proceeds; if it fails or times out, the system must decide whether to retry, fall back to the original output, or escalate for human review. For high-throughput pipelines, batch multiple outputs into a single request where the model processes each independently, but be aware that cross-output context leakage is possible if the model confuses document boundaries.

Streaming-compatible variant: When your primary model streams tokens, you cannot wait for the full output before deduplication. Implement a sliding window buffer that accumulates sentences as they complete (detected by terminal punctuation or newline boundaries). For each new sentence, check it against a local hash set of normalized prior sentences from the same stream. If it matches a verbatim duplicate, suppress it from the emitted stream and log the event. Only invoke the LLM-based stripping prompt for edge cases where near-duplicate or semantically equivalent sentences require judgment—such as intentional repetition for emphasis, refrains in creative text, or restated conclusions. This hybrid approach keeps latency low for the common case while reserving model intelligence for ambiguous decisions.

Validation and retry logic: After the stripping prompt returns, validate the output against a simple contract: (1) the response must contain the expected output schema fields, (2) the cleaned_text must not be empty if the input was non-empty, (3) the removed_sentences count should be zero or positive, and (4) the cleaned_text must be shorter than or equal to the input length. If validation fails, retry once with the same prompt and input. If the second attempt also fails, log the failure with the input hash, original output, and validator error details, then fall back to the original un-stripped text with a dedup_failed flag attached to the response metadata. For high-risk domains like legal or medical documentation, route validation failures to a human review queue instead of silently falling back.

Model choice and cost control: This repair task is structurally simple—sentence boundary detection, string comparison, and basic semantic judgment—so you can use a smaller, faster, and cheaper model than your primary generation model. A capable small model like Claude Haiku, GPT-4o-mini, or an open-weight 7B-13B parameter model is usually sufficient. If your primary output is already expensive to generate, adding a second large-model call for deduplication doubles cost unnecessarily. For batch pipelines, consider running the stripping prompt on a cheaper model first, then only escalating ambiguous cases to a larger judge model when the confidence score from the smaller model falls below your threshold.

Observability and production monitoring: Instrument the harness with metrics that matter: dedup_request_count, dedup_success_rate, avg_sentences_removed_per_doc, dedup_latency_p50/p99, validation_failure_rate, and fallback_rate. Track the distribution of removed sentence counts—a sudden spike may indicate your primary model has started repeating itself due to a prompt regression, temperature change, or context window exhaustion. Log a sample of removed sentences indexed by input hash so you can spot-check whether the stripping prompt is being too aggressive (removing intentional repetition) or too conservative (missing obvious duplicates). Set alerts on fallback_rate exceeding 5% or validation_failure_rate exceeding 2% over a rolling window.

What to avoid: Do not use this prompt as a substitute for fixing the root cause of repetition in your primary generation prompt. If your model consistently produces duplicate sentences, investigate temperature settings, repetition penalties, prompt instructions, or context window overstuffing before relying on a repair step. Do not apply this prompt to outputs where sentence-level repetition is semantically meaningful, such as poetry, song lyrics, legal boilerplate that must appear verbatim, or instructional content where repetition is pedagogical. In those cases, either skip the stripping step entirely or configure a whitelist of allowed repeated phrases that the prompt should preserve.