Pre-Execution Deduplication Check Prompt

This prompt is designed for workflow orchestration engineers and AI platform builders who need to insert a guardrail immediately before an agent is dispatched to perform a task. Its primary job is to prevent wasted compute and reconciliation debt by checking a pending task against a registry of recently completed and currently in-flight tasks. The ideal user is someone building a task router, a multi-agent dispatcher, or an orchestration layer where the cost of duplicate execution is high—either in dollars, data mutation conflicts, or user experience degradation.

Use this prompt when you have a structured record of active and recent workstreams and a new task payload ready for assignment. The prompt requires, at minimum, a [PENDING_TASK] description and a [WORKSTREAM_REGISTRY] containing task summaries, statuses, and unique identifiers. It is most effective when the registry is a machine-readable list of objects, not a free-text log. The prompt performs a semantic comparison to determine if the new task is a duplicate of an existing one, returning a structured decision with a confidence score and a reference to the matched workstream. It is not a replacement for exact-match idempotency keys; it is a semantic safety net for catching near-duplicates that slip past deterministic checks.

Do not use this prompt for real-time, sub-millisecond dispatch loops where the latency of an LLM call is unacceptable. It is also unsuitable as the sole deduplication mechanism for financial transactions or other strictly ordered operations where an idempotency key and database constraint are mandatory. In high-risk domains like healthcare or finance, the prompt's output should be treated as a recommendation that requires a human-in-the-loop review or a deterministic fallback before blocking a task. Finally, this prompt is not designed to detect conflicts across tenant boundaries in multi-tenant systems; for that, pair it with a tenant-isolation check in the application layer.

This prompt template is the core of the pre-execution deduplication check. It is designed to be inserted into an orchestration layer immediately before an agent is dispatched. The prompt receives a pending task payload and a registry of recently completed and in-flight tasks, then returns a structured decision: proceed, block as duplicate, or flag for human review. The square-bracket placeholders let you swap in your own task schema, deduplication rules, and risk thresholds without rewriting the instruction logic.

textCopy
You are a deduplication guard for an agent orchestration system. Your job is to prevent duplicate work before compute is spent.

You will receive a [PENDING_TASK] and a [TASK_REGISTRY] containing recently completed and in-flight tasks. Compare the pending task against the registry using semantic matching, not just keyword overlap.

[MATCHING_RULES]
- Two tasks are duplicates if they share the same core intent, primary entities, and desired outcome, even if phrased differently.
- A task is a partial duplicate if it overlaps significantly in scope but differs in parameters. Flag partial duplicates for review.
- Ignore differences in timestamp, request ID, or formatting.
- Respect [TENANT_ISOLATION_RULES]: never match tasks across tenant boundaries.

[OUTPUT_SCHEMA]
Return a single JSON object with these fields:
{
  "decision": "proceed" | "block" | "review",
  "confidence": 0.0-1.0,
  "matched_task_id": string | null,
  "match_type": "exact_duplicate" | "partial_overlap" | "none",
  "rationale": string,
  "recommended_action": string
}

[CONSTRAINTS]
- If confidence is below [MIN_CONFIDENCE_THRESHOLD], default to "review".
- If the pending task has an explicit [IDEMPOTENCY_KEY], check it against the registry's idempotency keys first.
- Do not fabricate matches. When in doubt, flag for review rather than blocking incorrectly.
- If the registry is empty or unavailable, return decision "proceed" with confidence 0.0 and note the missing data.

[EXAMPLES]
Example 1 - Exact duplicate:
Pending: {"intent": "summarize quarterly earnings for Q3 2024", "entities": ["Acme Corp"], "output_format": "executive brief"}
Registry: [{"id": "task-42", "intent": "summarize Q3 2024 earnings for Acme Corp", "status": "completed"}]
Output: {"decision": "block", "confidence": 0.97, "matched_task_id": "task-42", "match_type": "exact_duplicate", "rationale": "Same intent, entity, and output scope. Completed task covers this request.", "recommended_action": "Return cached result from task-42."}

Example 2 - Partial overlap:
Pending: {"intent": "analyze customer churn for enterprise segment", "entities": ["enterprise"], "time_range": "Q3 2024"}
Registry: [{"id": "task-38", "intent": "analyze customer churn across all segments", "entities": [], "time_range": "Q3 2024", "status": "in_flight"}]
Output: {"decision": "review", "confidence": 0.72, "matched_task_id": "task-38", "match_type": "partial_overlap", "rationale": "In-flight task covers a superset of the requested scope. Enterprise segment analysis may be derivable from the broader task.", "recommended_action": "Check if task-38 can satisfy the enterprise-specific request before dispatching a new agent."}

Now evaluate the pending task against the registry and return only the JSON object.

[PENDING_TASK]
[PENDING_TASK_PAYLOAD]

[TASK_REGISTRY]
[TASK_REGISTRY_PAYLOAD]

To adapt this template, replace the placeholder payloads with your actual task and registry data structures. The [MATCHING_RULES] section should reflect your organization's definition of duplicate work—tighten or loosen the semantic matching criteria based on your tolerance for false positives versus wasted compute. The [OUTPUT_SCHEMA] is the contract your orchestration layer will parse; keep the field names stable so downstream routing logic doesn't break. If your system uses idempotency keys, wire them into the [CONSTRAINTS] check as a fast path before running the full semantic comparison. For multi-tenant platforms, the [TENANT_ISOLATION_RULES] placeholder must be replaced with your actual tenant boundary logic to prevent cross-tenant data leakage through the deduplication check itself.

Before shipping this prompt, test it against a golden dataset of known duplicates, near-duplicates, and distinct tasks. Measure both false-positive blocks (which frustrate users by delaying legitimate work) and false-negative passes (which waste compute on redundant execution). If your task registry can grow large, add a retrieval step before this prompt to pre-filter the registry to candidate matches, keeping the prompt context within token limits. For high-stakes workflows where a false block could delay critical operations, route all "block" decisions with confidence below 0.95 to a human review queue rather than automatically rejecting them.

This prompt is designed to sit directly in the critical path of an agent task dispatcher. Before a task is assigned to a worker agent, the orchestrator calls this prompt with the pending task payload and a snapshot of the active workstream registry. The model returns a structured decision (DUPLICATE, UNIQUE, or AMBIGUOUS) along with a confidence score and a reference to the conflicting task ID. The orchestrator should treat this as a blocking gate: DUPLICATE results must halt dispatch and either discard the task or queue it for human review, while AMBIGUOUS results should trigger a secondary check or a short time-to-live (TTL) hold to see if the in-flight task completes.

To integrate this into a production harness, wrap the prompt call in a function with a strict timeout (e.g., 500ms for fast models, 2s for slower ones) because a deduplication check that takes longer than the task itself defeats the purpose. On timeout, the system should default to a safe fallback—either AMBIGUOUS with a log event or a configurable policy like 'allow dispatch but flag for post-hoc review.' Validate the model's output against a JSON schema that requires decision (enum of DUPLICATE, UNIQUE, AMBIGUOUS), confidence (float 0-1), matched_task_id (string or null), and rationale (string). If the output fails schema validation, retry once with a stricter prompt that includes the validation error. If it fails again, escalate to AMBIGUOUS and log the raw response for debugging.

For the workstream registry input, provide the model with a structured list of active and recently completed tasks (last N minutes, configurable by your SLA). Each entry should include a task ID, a semantic fingerprint or summary, the assigned agent, the status (IN_FLIGHT, COMPLETED, FAILED), and a timestamp. Avoid sending raw, unstructured task payloads; preprocess them into a canonical fingerprint using a companion prompt like the Agent Workstream Fingerprint Extraction Prompt. This keeps the deduplication prompt's context window small and its latency predictable. Log every decision—including the model, the fingerprint, the matched task, and the confidence score—to an append-only audit table. This log becomes the evidence trail for tuning the deduplication threshold and for diagnosing false positives that blocked legitimate work.

Model choice matters here. This prompt requires strong semantic comparison capabilities, so prefer models with high performance on sentence similarity and entailment tasks. For high-throughput pipelines, consider using a smaller, fine-tuned classifier model for the initial check and only escalating to a larger LLM when the classifier's confidence falls in a gray zone. If you're using a cloud LLM API, implement client-side retries with exponential backoff for transient errors (429, 503) but do not retry on schema validation failures more than once. Finally, build a lightweight eval harness that replays known duplicate and unique task pairs through the prompt weekly, measuring precision and recall against your labeled dataset. When the false-positive rate exceeds your operational threshold (e.g., 5%), investigate whether the fingerprint extraction step or the deduplication prompt itself needs tuning.