Batch Processing Deduplication Harness Prompt

This prompt is designed for platform engineers and MLOps teams running large-scale model inference pipelines where a single batch job produces multiple outputs containing duplicate records. The core job-to-be-done is consolidating redundant data before it hits downstream ingestion, reporting, or API delivery. It executes a two-pass deduplication strategy: first, it removes duplicate entries within each individual model output (intra-output dedup). Second, it selects a single canonical record across all outputs in the batch (cross-output canonical selection). This is not a real-time, single-request deduplication tool, and it is not designed for training data curation where diversity preservation is the primary goal.

To use this effectively, you must provide the full batch of model outputs as a structured input, along with a clear identity key that defines what makes two records the 'same.' The harness expects idempotency keys for each batch job to prevent double-processing, and it should be wired into a post-processing step that can track progress and recover from partial failures. For example, if you are running 10,000 inference calls that each return a list of extracted entities, this prompt can reduce the final payload from 150,000 records to 80,000 unique canonical entities. Do not use this prompt when you need to preserve all variant phrasings for human review, or when the cost of accidentally merging two distinct but similar records is unacceptable.

Before wiring this into production, define your evaluation criteria: measure the deduplication ratio, check for false merges against a labeled sample, and ensure no records are lost due to a bug in the cross-output selection logic. If your use case involves regulated data, always include a human review step for records flagged as ambiguous matches. The next section provides the copy-ready prompt template you can adapt and test immediately.

This prompt template implements the two-pass deduplication strategy for batch processing pipelines. It expects you to have already completed individual model calls and collected their outputs into a structured batch. The prompt first performs intra-output deduplication on each record, then executes cross-output canonical record selection across the entire batch. Use this template as the core instruction set inside your batch processing harness, wiring it after individual model calls complete and before results enter downstream systems.

textCopy
You are a batch deduplication engine operating on a collection of model outputs. Your job is to remove duplicate and redundant content in two passes, producing a clean, canonical result set with full traceability.

## INPUT
You will receive a batch of model outputs in the following structure:
[BATCH_INPUT]

Each output has:
- An `output_id` (unique identifier)
- A `payload` (the model's raw output, which may be text, JSON, or structured records)
- Optional `metadata` (generation timestamp, model version, confidence scores)

## PASS 1: INTRA-OUTPUT DEDUPLICATION
For each individual output in the batch:
1. Identify duplicate or near-duplicate entries within that single output's payload.
2. Apply the deduplication strategy specified in [INTRA_DEDUP_STRATEGY].
3. Preserve the first occurrence unless [PRIORITY_RULES] specify otherwise.
4. Record removed entries in a `removed_intra` list with the reason for removal.

## PASS 2: CROSS-OUTPUT CANONICAL RECORD SELECTION
After all outputs have been intra-deduplicated:
1. Compare records across all outputs in the batch.
2. Identify records that represent the same entity, fact, or claim using [SIMILARITY_THRESHOLD] and [MATCH_FIELDS].
3. For each duplicate cluster, select a single canonical record using [CANONICAL_SELECTION_RULE].
4. Record all non-canonical records in a `removed_cross` list with the canonical record they map to.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
[OUTPUT_SCHEMA]

## CONSTRAINTS
[CONSTRAINTS]

## IDEMPOTENCY
Use the provided [IDEMPOTENCY_KEY] to track this deduplication run. If a run with this key has already been completed, return the cached result instead of re-processing.

## PROGRESS TRACKING
Report progress at these checkpoints:
- After intra-deduplication of every [PROGRESS_INTERVAL] outputs
- After cross-output clustering is complete
- Before final output emission

## PARTIAL FAILURE RECOVERY
If processing fails on a subset of outputs:
1. Complete deduplication on all successful outputs.
2. Return a `failed_output_ids` array with the IDs that could not be processed.
3. Include a `recovery_hint` for each failure suggesting whether retry, skip, or human review is appropriate.

## EXAMPLES
[EXAMPLES]

Adapt this template by replacing each square-bracket placeholder with your pipeline's specific configuration. [BATCH_INPUT] should contain the actual serialized batch of model outputs. [INTRA_DEDUP_STRATEGY] specifies whether to use exact matching, fuzzy matching with a threshold, or semantic similarity. [CANONICAL_SELECTION_RULE] defines how to pick the winner from a duplicate cluster—options include keep-longest, keep-highest-confidence, keep-most-recent, or a custom priority function. [OUTPUT_SCHEMA] must be a strict JSON Schema definition that your downstream systems can validate against. [CONSTRAINTS] should include any domain-specific rules, such as field-level uniqueness requirements or mandatory preservation of certain record types. [IDEMPOTENCY_KEY] should be a deterministic hash of the batch content and configuration to enable safe retries without double-processing. [PROGRESS_INTERVAL] controls how frequently the harness emits progress updates, typically every 10 or 100 outputs depending on batch size.

Before deploying this prompt into production, validate it against a golden dataset of known duplicates with expected canonical outputs. Test edge cases including empty batches, single-output batches, batches where every record is a duplicate, and batches with conflicting priority rules. Wire the output through a JSON Schema validator before allowing it into downstream systems. For high-stakes pipelines where duplicate removal could cause data loss, configure the harness to require human review for clusters above a configurable ambiguity threshold. Do not use this prompt for real-time, single-request deduplication—it is designed for batch processing where the overhead of two-pass analysis is acceptable.

The Batch Processing Deduplication Harness Prompt is designed to operate as a post-inference repair step within a larger batch processing pipeline. It expects a collection of model outputs—each already validated for structural integrity—and performs a two-pass deduplication: first within each individual output, then across the entire batch to select canonical records. This harness should be invoked after all model calls in a batch have completed and passed their individual schema validators, but before the results are written to the final data store or served to downstream consumers. The prompt itself is stateless; all state management, retry logic, and progress tracking must be handled by the application layer wrapping it.

To integrate this prompt, wrap it in a function that accepts a batch of records with unique [OUTPUT_ID] values and an [IDEMPOTENCY_KEY] derived from the batch hash or job ID. Before calling the model, check a persistent store (such as a database or cache) for a completed result under that idempotency key. If found, return the cached result immediately. If not, call the model with the full batch payload, validate the response against the expected output schema (a list of canonical records with source output IDs and deduplication rationale), and store the result keyed by the idempotency key. For large batches that exceed context windows, split the batch into overlapping windows of [WINDOW_SIZE] records, process each window, then run a final cross-window deduplication pass using the same prompt with the merged intermediate results. Log every invocation with the batch size, window count, model used, latency, and deduplication ratio for observability.

Partial failure recovery is critical for production use. If the model call fails, times out, or returns an unparseable response, implement a retry strategy with exponential backoff (starting at 1 second, max 3 retries). If retries are exhausted, fall back to a degraded mode: return the original outputs with a deduplication_status: failed flag and preserve all records rather than risk data loss. For batches where the model response is valid but the deduplication appears suspicious—such as a deduplication ratio above [MAX_EXPECTED_RATIO] or below [MIN_EXPECTED_RATIO]—route the batch to a human review queue with the before/after counts and a sample of the deduplication decisions. Never silently drop records without an audit trail. The harness should emit structured logs for each deduplication decision, including the kept record, the removed record, the match type (exact, fuzzy, semantic), and the confidence score, so that downstream audit systems can reconstruct the full decision history.