Duplicate Key Resolution Prompt

This prompt is for data pipeline engineers and integration developers who consume structured output (JSON, YAML) from an LLM and encounter duplicate keys in generated objects. The job is to resolve these duplicates deterministically—by applying a consistent strategy such as first-wins, last-wins, merge, or error-flagging—before the malformed payload reaches a downstream parser, database insert, or API call. The ideal user is someone building a production extraction or generation harness where JSON.parse succeeds but the resulting object violates application-level uniqueness constraints on keys.

Use this prompt when your validation layer detects duplicate keys in an otherwise parseable structure and you need the model to apply a resolution rule and return a clean, schema-compliant object. The prompt requires you to supply the malformed input, the target schema, and the chosen resolution strategy. It is not a replacement for a deterministic post-processing script when the rule is trivial (e.g., always take the last occurrence). Reserve it for cases where the resolution requires semantic judgment—for example, merging two partial address blocks under the same key, or deciding which of two conflicting status fields is more consistent with surrounding context.

Do not use this prompt for real-time, latency-sensitive ingestion where a sub-100ms fix is required; a hardcoded resolver in your application code will be faster and cheaper. Avoid it when the duplicate key indicates a deeper failure in the upstream prompt that should be fixed at the source rather than patched in a repair loop. If the output contains personally identifiable information, financial records, or clinical data, ensure the repair step runs in a logged, auditable environment with human review gating before the corrected payload is persisted. The next section provides the copy-ready prompt template you can adapt and wire into your validation-and-repair harness.

This prompt template is designed to be dropped into a post-processing or repair step when a model's structured output (JSON, YAML, or XML) contains duplicate keys that violate the target schema. It instructs the model to detect duplicates and apply a consistent resolution strategy—first-wins, last-wins, merge, or error-flagging—based on a configurable [RESOLUTION_STRATEGY] parameter. The prompt uses few-shot examples to demonstrate each strategy, making it more reliable than relying on procedural instructions alone.

textCopy
You are a structured data repair agent. Your task is to resolve duplicate keys in the provided [INPUT_FORMAT] document.

**Resolution Strategy:** [RESOLUTION_STRATEGY]
**Output Schema:** [OUTPUT_SCHEMA]
**Constraints:** [CONSTRAINTS]

**Strategy Definitions:**
- `first-wins`: Keep the first occurrence of a duplicate key and discard all subsequent occurrences.
- `last-wins`: Keep the last occurrence of a duplicate key and discard all earlier occurrences.
- `merge`: Combine values from duplicate keys into an array, preserving all values in order of appearance.
- `error-flag`: Do not resolve duplicates. Instead, return an error object listing all duplicate keys and their locations.

**Examples:**

[EXAMPLE_INPUT_1]
[EXAMPLE_OUTPUT_1]

[EXAMPLE_INPUT_2]
[EXAMPLE_OUTPUT_2]

[EXAMPLE_INPUT_3]
[EXAMPLE_OUTPUT_3]

**Input to Repair:**
[INPUT]

Return ONLY the repaired [INPUT_FORMAT] document with no additional commentary. If using `error-flag`, return a JSON error object with keys `error_type`, `duplicate_keys`, and `locations`.

To adapt this template, replace the square-bracket placeholders with concrete values. [RESOLUTION_STRATEGY] must be exactly one of the four defined strategies. [OUTPUT_SCHEMA] should be a JSON Schema or a natural-language description of the expected structure. [CONSTRAINTS] can include field-specific rules like "the id field must remain unique" or "array fields should be concatenated, not overwritten." The [EXAMPLE_INPUT_] and [EXAMPLE_OUTPUT_] pairs are critical: provide at least two examples that match your expected data shape and chosen strategy. For high-risk data pipelines, add a validation step after repair that checks for key uniqueness and schema compliance before the output enters downstream systems. If the repair fails validation, escalate for human review rather than retrying indefinitely.

The duplicate key resolution prompt is designed to sit inside a data ingestion or transformation pipeline where structured outputs (JSON, YAML) from upstream models or parsers may contain conflicting keys. The harness should invoke this prompt as a post-processing step, not as part of the primary generation call. This separation keeps the main prompt focused on content quality while the repair prompt handles structural conflicts. The typical integration point is immediately after JSON.parse succeeds but before the object is written to a database, cache, or downstream consumer. The harness must supply the malformed object as [INPUT], the desired resolution strategy (first-wins, last-wins, merge, or error-flag) as [CONSTRAINTS], and the expected output schema as [OUTPUT_SCHEMA].

Validation is the critical first layer of the harness. Before calling the prompt, confirm that the input actually contains duplicate keys by parsing the raw string into an intermediate representation that surfaces collisions—standard JSON.parse silently drops duplicates, so use a streaming parser like json-stream or a custom tokenizer that detects repeated keys at the same nesting level. After the model returns a repaired object, run it through a structural validator that checks: (1) no duplicate keys remain at any depth, (2) the output matches [OUTPUT_SCHEMA] exactly, and (3) the resolution strategy was applied consistently (e.g., if first-wins was specified, verify that the first occurrence's value was preserved). If validation fails, retry once with the validation error message appended to [CONTEXT]. After two failures, log the original input, the model's attempts, and the validation errors, then route to a human review queue or dead-letter topic. For high-throughput pipelines, add a circuit breaker that stops retries if the failure rate exceeds 5% in a rolling window.

Model choice and latency budgeting matter here. This is a structured repair task with low ambiguity, so smaller and faster models (e.g., Claude Haiku, GPT-4o-mini, or fine-tuned open-weight models) often perform as well as large models at a fraction of the cost and latency. Set a strict timeout of 2-3 seconds per call. If the pipeline is processing streaming data, batch up to 10 malformed objects per prompt call to amortize overhead, but keep each object's repair independent—do not let the model cross-contaminate resolutions between objects. Log every repair attempt with: input hash, resolution strategy, model ID, latency, token count, validation result, and the final output. This audit trail is essential for debugging resolution inconsistencies and for demonstrating data lineage to downstream consumers who may question why a value changed. Avoid using this prompt for objects where key uniqueness is semantically meaningful (e.g., JSON-LD contexts, cryptographic key material, or legal document identifiers)—in those cases, duplicate keys should always escalate to a human rather than be silently resolved.