Output Deduplication Audit Trail Prompt

This prompt is for governance, compliance, and data engineering teams who need a verifiable record of every deduplication decision made by an AI system. It is not a prompt for performing deduplication faster or cheaper; it is a prompt for making deduplication decisions traceable and defensible. The ideal user is an engineer or auditor responsible for a data pipeline where silent data loss is unacceptable and every record merge or removal must be explainable. You should use this prompt when your system of record requires an immutable change history, when you are preparing for an external audit, or when you need to prove that no data was silently dropped during a batch normalization process.

The prompt produces a structured, machine-readable audit log that captures the before state, after state, decision rationale, and a unique event identifier for each duplicate detected and resolved. This output is designed to be ingested by an audit system or stored as an append-only log. The required context includes the original list of records, the deduplication rules or priority logic applied, and a unique batch identifier to tie the audit trail back to a specific processing event. Do not use this prompt when you are simply cleaning data for a non-critical internal dashboard, when the volume of duplicates is so high that an audit trail would be cost-prohibitive to store, or when the deduplication logic is a simple exact-match removal that can be fully captured by a deterministic function in application code.

Before implementing this prompt, ensure your pipeline can handle the additional latency and token cost of generating a detailed rationale for every duplicate pair. The primary failure mode is a bloated audit log that becomes too expensive to store and query, so consider sampling or threshold-based logging for low-risk deduplication events. The next step after generating the audit trail is to validate its structural integrity against the [OUTPUT_SCHEMA] and to write it to an append-only store with the batch identifier as the primary key. Avoid treating this prompt as a replacement for deterministic deduplication logic; it is a governance wrapper that should sit around an existing deduplication process to make its decisions transparent.

This prompt template is designed to be wired directly into your deduplication pipeline. It accepts a list of candidate records and a set of deduplication rules, then produces a structured audit log. Each entry in the log captures the duplicate pair, the decision made, the rationale, and the before/after state. The output is a JSON array of audit entries, suitable for ingestion into your existing governance, compliance, or logging infrastructure. Use this when you need traceable, explainable deduplication—not just a cleaned dataset.

textCopy
You are an audit-grade deduplication engine. Your task is to analyze a list of input records, identify duplicates according to the provided rules, and produce a structured audit trail for every decision.

## INPUT RECORDS
[INPUT_RECORDS]

## DEDUPLICATION RULES
[DEDUPLICATION_RULES]

## OUTPUT SCHEMA
Return a JSON object with a single key "audit_trail" containing an array of audit entries. Each entry must conform to this schema:
{
  "entry_id": "string, unique identifier for this audit entry",
  "timestamp": "string, ISO 8601 timestamp of the decision",
  "decision": "MERGE | DELETE | KEEP_ORIGINAL | FLAG_FOR_REVIEW",
  "rationale": "string, concise explanation of why this decision was made, referencing the specific rule applied",
  "duplicate_group_id": "string, identifier linking records involved in the same duplicate cluster",
  "records_involved": [
    {
      "record_id": "string, the original identifier from the input",
      "role": "KEPT | DISCARDED | MERGED_INTO",
      "data_before": {},
      "data_after": {}
    }
  ],
  "rule_applied": "string, the specific rule from DEDUPLICATION_RULES that triggered this decision",
  "confidence": "float between 0.0 and 1.0"
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Generate the audit trail now.

To adapt this template, replace the square-bracket placeholders with your specific data and configuration. [INPUT_RECORDS] should be your serialized list of candidate records, each with a unique identifier. [DEDUPLICATION_RULES] defines the logic for what constitutes a duplicate—be explicit about field comparisons, similarity thresholds, and precedence rules. [CONSTRAINTS] is where you enforce immutability requirements, such as 'never modify the original record data' or 'log all decisions, even no-op ones.' [EXAMPLES] should include one or two few-shot demonstrations of correct audit entries for your domain. [RISK_LEVEL] controls the model's behavior: for high-risk domains like healthcare or finance, set this to 'HIGH' and include a constraint that ambiguous cases must use the FLAG_FOR_REVIEW decision. After generating the audit trail, validate that every input record is accounted for in the output and that no data was silently altered.

This prompt is not a standalone deduplication tool; it is a governance component designed to produce a structured, machine-readable audit trail. The primary integration point is a post-processing step in a data pipeline, positioned immediately after a deduplication action has been performed by another system or prompt. Its job is to take the 'before' and 'after' state of a record set and generate a traceable log of every decision made. The output is a JSON array of decision objects, each with a unique decision_id, a rationale, and a before/after snapshot. This audit log should be treated as an append-only record in your system of record.

To wire this into an application, you must provide the [ORIGINAL_RECORDS] and [DEDUPLICATED_RECORDS] arrays as input. The application layer is responsible for computing the diff or providing both states; this prompt does not perform the deduplication itself. Validation is critical: before writing the prompt's output to your audit database, validate that every decision_id is unique, that each before record exists in the original set, and that each after record exists in the deduplicated set. A strict JSON Schema validation should reject any audit entry that references records not present in the provided inputs. For high-compliance environments, implement a two-phase commit: first, write the audit log to a staging table, run the validation, and only then append it to the immutable audit ledger.

For model choice, prefer a deterministic or low-temperature setting (e.g., temperature=0) to maximize consistency in the rationale field. The decision_id should be a UUID generated by the model, but you must implement a retry-and-replace mechanism in the application harness: if the model generates a duplicate ID, re-prompt for that specific entry or generate the UUID in the application layer and inject it. Logging is essential: capture the full prompt, the raw model response, the validation result (pass/fail), and the final written audit entries. This ensures that even a malformed audit trail can be reconstructed. Avoid using this prompt for real-time, user-facing deduplication; it is designed for asynchronous, batch-oriented governance workflows where traceability is the primary requirement, not speed.