Memory Deduplication Prompt for Redundant Facts

This prompt is designed for knowledge engineers and infrastructure teams maintaining shared fact stores, blackboard systems, or agent-accessible memory across multi-agent systems. The core job-to-be-done is cleaning a structured memory store of redundant, near-duplicate, or semantically equivalent facts that accumulate when multiple agents write independently. It is not a general-purpose deduplication tool for unstructured text or document corpora. It assumes facts are already extracted and stored as structured entries with source attribution, a confidence score, and a unique identifier. The ideal user is an AI platform engineer or knowledge infrastructure operator who needs to reduce memory bloat, prevent conflicting beliefs, and maintain a single source of truth for downstream agents.

Use this prompt when your fact store exhibits symptoms of redundancy: multiple entries asserting the same core claim with minor wording differences, facts from different sources that are logically equivalent, or outdated entries that should be merged into a newer, higher-confidence version. The prompt produces structured merge or delete operations with explicit equivalence justifications, source provenance tracking, and confidence scores. It is designed to be wired into an automated pipeline where a batch of candidate facts is periodically reviewed for deduplication. The output is a set of deterministic operations—merge, delete, or keep—that can be applied programmatically to your store. A critical implementation detail is that this prompt requires a pre-filtered candidate set; feeding it the entire memory store at once will degrade performance and increase the risk of false merges. Instead, use a retrieval step to group facts by entity or topic before invoking this prompt.

Do not use this prompt for deduplicating raw documents, user-generated text, or conversational logs. It is not designed for fuzzy string matching across paragraphs of prose. It will fail if the input facts lack structured fields like fact_id, source_agent, confidence, or timestamp. It is also inappropriate for real-time, latency-sensitive agent loops; this is a batch hygiene operation, not a synchronous step in an agent's critical path. If your system requires real-time deduplication on write, implement a deterministic hash-based or embedding-distance check at ingestion time and reserve this prompt for periodic deep cleaning. Finally, this prompt does not resolve contradictory facts where both claims are plausible but incompatible—for that, use the Agent Belief Revision Prompt on Conflicting Evidence. The next step after understanding this boundary is to prepare your candidate fact set and review the prompt template for required placeholders.

This prompt template is designed to be dropped directly into an agent memory maintenance pipeline. It expects a batch of fact entries from your shared memory store and produces a set of merge or delete operations for redundant or near-duplicate facts. The template uses square-bracket placeholders that you must replace with your actual data, schemas, and constraints before execution. Every placeholder is documented so you can wire it into your existing memory store API, validation layer, and audit trail.

textCopy
You are a memory deduplication agent responsible for cleaning a shared agent memory store. Your task is to identify redundant, duplicate, or near-duplicate fact entries and produce merge or delete operations with clear equivalence justifications.

## INPUT
You will receive a batch of fact entries from the shared memory store. Each entry has a unique ID, a fact statement, source attribution, a confidence score, and a timestamp.

[INPUT]

## OUTPUT SCHEMA
Return a JSON object with an "operations" array. Each operation must conform to this schema:
{
  "operations": [
    {
      "action": "merge" | "delete" | "keep",
      "primary_id": "string (the surviving fact ID for merge, or the fact ID for delete/keep)",
      "duplicate_ids": ["string (list of IDs to merge into primary or to delete)"],
      "merged_fact": "string (the consolidated fact statement, required for merge actions)",
      "equivalence_justification": "string (explain why these facts are equivalent, citing semantic overlap, contradiction, or temporal supersession)",
      "confidence_impact": "string (how the merge or delete affects overall confidence in the surviving fact: increased, decreased, unchanged)"
    }
  ],
  "unchanged_ids": ["string (list of fact IDs that require no action)"]
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## INSTRUCTIONS
1. Compare all facts pairwise for semantic equivalence, not just string similarity. Two facts are equivalent if they convey the same information even with different wording.
2. A fact is a near-duplicate if it adds no new information beyond what another fact already states, even if the wording differs.
3. When merging, produce a consolidated fact that preserves all unique information from the merged entries. Do not drop details that appear in only one of the duplicates.
4. If two facts contradict each other, flag the contradiction in the equivalence justification and prefer the fact with the higher confidence score and more recent timestamp. Do not silently merge contradictory facts.
5. If confidence scores differ across duplicates, the merged fact should reflect the highest confidence score and note the discrepancy.
6. If you are uncertain whether two facts are truly equivalent, default to "keep" for both and explain your uncertainty in the justification.
7. Source attributions must be preserved in the merged fact. If multiple sources contributed, list all of them.
8. Do not merge facts that refer to different entities, time periods, or contexts, even if the surface wording is similar.

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace each square-bracket placeholder with your actual data and rules. [INPUT] should contain your batch of fact entries serialized as JSON, with fields matching your memory store schema. [CONSTRAINTS] should specify domain-specific rules such as maximum batch size, forbidden merge categories, or required confidence thresholds. [EXAMPLES] should include 2–4 few-shot examples showing correct merge, delete, and keep decisions with justifications that match your domain's equivalence standards. [RISK_LEVEL] should be set to high, medium, or low to control whether the system requires human review before applying operations. For high-risk domains such as healthcare or finance, always set RISK_LEVEL to high and route all merge and delete operations to a human approval queue before writing back to the memory store. Validate the output against the schema before applying any operation, and log every deduplication decision with the original input batch ID for auditability.

This prompt is designed to be called as a batch deduplication step within a shared memory maintenance pipeline, not as a real-time user-facing operation. The typical integration pattern is a scheduled job or a trigger fired after a configurable number of new memory writes. The caller assembles a batch of candidate memory entries—usually those sharing overlapping entity keys, time windows, or embedding clusters—and passes them as [MEMORY_ENTRIES]. The prompt returns a structured list of merge or delete operations, each with an equivalence justification. The application layer should never auto-apply these operations without validation.

Validation and safety checks are mandatory. Before applying any merge or delete operation, validate that the returned entry_ids exist in the current memory store and that no entry has been modified since the batch was assembled (optimistic concurrency check). Reject any operation where the equivalence_justification is empty, fewer than 10 words, or contains only generic phrases like 'they are the same.' For high-stakes knowledge bases, route all merge operations with confidence below 0.85 to a human review queue and auto-apply only delete operations where the target entry is a strict substring or older timestamp duplicate of the surviving entry. Log every applied and rejected operation with the full prompt payload, model response, validator results, and reviewer identity for auditability.

Model choice and retry strategy matter here. Use a model with strong structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent) and enforce the [OUTPUT_SCHEMA] via constrained generation or strict JSON mode. Set temperature=0 to minimize variance in equivalence judgments. If the response fails schema validation, retry once with the validation errors injected into [CONSTRAINTS] as explicit prohibitions. If the second attempt also fails, log the failure and escalate the entire batch to human review rather than silently dropping entries. For large memory stores, partition batches by entity cluster (e.g., same entity_id or document_source) and process partitions in parallel with a merge worker that resolves cross-partition conflicts by preferring the operation with the higher confidence score.

What to avoid: Do not use this prompt on memory entries that represent fundamentally different fact types (e.g., a user preference and a document summary) even if they share keywords—the prompt's similarity detection is semantic, not taxonomic, and will produce false merges. Do not run deduplication on actively written memory without a lock or lease on the affected entries. Do not treat the model's confidence score as a calibrated probability; use it only for relative ranking within a batch. Finally, always retain the original entries for at least one retention period after a merge so that incorrect deduplications can be rolled back with a state versioning and rollback prompt.