Streaming Output Deduplication Prompt Template

This prompt is designed for production engineers and streaming infrastructure developers who are consuming real-time LLM outputs and encountering content repetition across chunks. The core job-to-be-done is to programmatically clean a stream of text by removing duplicate sentences, paragraphs, or logical sections that the model has regenerated, while strictly preserving intentional repetition such as choruses in lyrics, repeated headers in structured lists, or deliberate refrains in generated prose. The ideal user is someone who already has a streaming ingestion pipeline and needs a deterministic or model-assisted deduplication step before the text is rendered to a user, written to a database, or passed to a downstream parser.

You should use this prompt when your streaming provider does not guarantee idempotent chunk delivery, when you observe models repeating the last sentence of a previous chunk at the start of the next one, or when your application logic for reassembling chunks introduces overlap. It is particularly effective when the output is natural language prose, markdown, or semi-structured text where semantic boundaries are clear. The prompt expects a pre-assembled text block or a sequence of chunks as input, along with a clear definition of what constitutes a duplicate in your domain. You must provide a [DEDUPLICATION_RULES] section specifying whether to match on exact string equality, fuzzy semantic similarity, or structural position, and whether certain patterns like numbered lists or code blocks are exempt from deduplication.

Do not use this prompt as a substitute for fixing the root cause of chunk overlap in your streaming client. If your chunk reassembly logic is producing overlapping windows due to a bug in your sliding buffer, fix that first. This prompt is also not appropriate for deduplicating records in a database, cleaning training data, or removing near-duplicate documents from a retrieval corpus—those are batch deduplication problems with different constraints. Avoid using this prompt when the output is a single, non-streamed completion, as the overhead of semantic comparison adds latency without benefit. Finally, if the content is highly structured JSON or code where duplication is a schema violation, prefer a schema-aware repair prompt from the Structured Output Repair pillar instead of this text-focused deduplication approach.

This prompt template is designed to be inserted into a post-processing step after streaming chunks have been assembled into a single candidate text. It instructs the model to act as a deduplication filter, identifying repeated sentences, paragraphs, or sections while preserving intentional repetition such as refrains, structured lists, or emphasis. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to integrate into an application harness where the assembled text, output schema, and deduplication rules are injected at runtime.

textCopy
You are a streaming output deduplication filter. Your task is to detect and remove unintentional duplicate content from the assembled streaming output provided below. Intentional repetition—such as poetic refrains, repeated headers in structured lists, emphasis through restatement, or template-driven repetition—must be preserved.

## INPUT
[ASSEMBLED_STREAMING_OUTPUT]

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
You must return a valid JSON object conforming to this schema:
{
  "deduplicated_output": "string (the cleaned text with duplicates removed)",
  "removed_segments": [
    {
      "text": "string (the duplicate text that was removed)",
      "reason": "exact_duplicate | near_duplicate | redundant_restatement",
      "location": "string (description of where the duplicate appeared relative to the original)"
    }
  ],
  "preserved_repetitions": [
    {
      "text": "string (the repeated text that was intentionally preserved)",
      "reason": "string (why this repetition was kept—e.g., refrain, list structure, emphasis)"
    }
  ],
  "deduplication_summary": "string (brief explanation of what was removed and why)"
}

## DEDUPLICATION RULES
1. Remove exact duplicate sentences, paragraphs, or multi-sentence blocks that appear more than once.
2. Remove near-duplicate content where the semantic meaning is identical but wording differs slightly (e.g., paraphrased restatements).
3. Preserve intentional repetition: refrains in songs or poems, repeated list headers, template boilerplate, emphasis through restatement, and call-and-response patterns.
4. When a duplicate is found, keep the first occurrence and remove subsequent occurrences.
5. Do not alter the content of non-duplicate text. Preserve original wording, punctuation, and formatting.
6. If no duplicates are found, return the original text unchanged with an empty removed_segments array.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Return ONLY the JSON object. Do not include any explanatory text outside the JSON.

To adapt this template for your application, replace each placeholder with concrete values. [ASSEMBLED_STREAMING_OUTPUT] should contain the full text assembled from streaming chunks before deduplication. [CONSTRAINTS] can specify domain-specific rules—for example, in a legal document context you might add 'Preserve numbered clause references even if repeated across sections.' [EXAMPLES] should include one or two few-shot demonstrations showing the model how to handle edge cases like near-duplicate detection versus intentional restatement. [RISK_LEVEL] should be set to 'low', 'medium', or 'high' to control the conservatism of the deduplication: at high risk levels, the model should err on the side of preserving potentially intentional repetition. Always validate the output JSON against the schema before passing deduplicated text downstream, and log all removed_segments for auditability.

The deduplication prompt is not a standalone module. It sits between the raw streaming accumulator and the downstream consumer—typically a renderer, database writer, or API response. In practice, you buffer streaming chunks until a termination signal (e.g., finish_reason='stop' or a closing SSE event), assemble the full text, and then invoke this prompt as a post-processing step. Do not run deduplication on partial streams; the model needs the complete context to distinguish intentional repetition (refrains, structured lists, repeated headers) from accidental duplication caused by overlapping chunks or model drift.

Wire the prompt into an async processing stage with a configurable timeout and a strict output contract. The prompt expects a [FULL_ASSEMBLED_TEXT] input and returns a JSON object with deduplicated_text and removed_segments fields. Validate the response against this schema immediately. If the output fails JSON parsing or schema validation, retry once with a stronger constraint instruction appended to the prompt. If the second attempt fails, log the raw assembled text and the failed response, then fall back to the original assembled text with a deduplication_failed flag set on the event. Never silently drop content.

For high-throughput systems, consider batching multiple assembled outputs into a single deduplication call if the combined token count stays within the model's context window. This reduces API overhead but increases blast radius on failure—a single malformed batch can corrupt multiple outputs. Implement per-item validation inside the batch response and isolate failures individually. For latency-sensitive streaming UIs, run deduplication as a non-blocking side effect: stream the raw assembled text to the user immediately, then patch the display if the deduplication step produces a cleaner version within a short window (e.g., 2 seconds).

Observability is critical because deduplication failures are often silent—the output looks plausible but contains repeated paragraphs that degrade user experience. Log the count of removed_segments per call and alert if the ratio of removed characters to total characters exceeds a threshold (e.g., 20%), which may indicate a broken assembly step upstream rather than a deduplication problem. Track the retry rate and schema validation failure rate separately. If the deduplication prompt consistently removes content that downstream consumers flag as false positives, collect those examples and add them as negative demonstrations in the [EXAMPLES] placeholder.

Model choice matters. Use a fast, inexpensive model for deduplication (e.g., a small Claude Haiku or GPT-4o-mini variant) because the task is pattern-matching heavy and does not require deep reasoning. Avoid using the same model instance that generated the original stream; a model that hallucinated repetition is more likely to miss its own errors. If the content domain includes regulated or high-stakes material, route deduplication outputs to a human review queue when the removed_segments list is non-empty, and always preserve the pre-deduplication text in your audit log alongside the cleaned version.