Agent Overlap Prevention in RAG Pipelines Prompt

This prompt is designed for AI engineers and platform operators who are building multi-agent retrieval-augmented generation (RAG) pipelines where multiple specialized retrieval agents operate in parallel over a shared document corpus. The core job-to-be-done is pre-retrieval scope negotiation: before any agent executes a vector or keyword search, this prompt acts as a coordination layer that assigns non-overlapping retrieval scopes to each agent. Without explicit coordination, agents frequently fetch overlapping document sets, wasting compute on redundant processing and creating reconciliation debt when outputs must be merged. The ideal user controls the orchestration layer that dispatches agent retrieval calls and can inject this coordination step between intent formation and actual retrieval execution. Required context includes a set of agent retrieval intents (what each agent plans to search for) and a shared document index snapshot (metadata, filters, or partition keys available in the retrieval system).

Concrete scenarios where this prompt applies: you have two or more retrieval agents that query the same corpus, retrieval latency or token costs from duplicate documents are measurable problems, and you can modify the orchestration layer to enforce the assigned scopes. For example, a legal research system might have one agent searching for case precedents and another searching for statutory interpretations—both hitting the same document store. Without coordination, both agents retrieve the same landmark cases, doubling embedding costs and forcing a downstream merge step to deduplicate citations. This prompt intervenes by analyzing the intents and index structure, then outputting mutually exclusive filter constraints (e.g., date ranges, document types, jurisdiction tags) that each agent must apply. The output is a structured scope assignment, not retrieval results—it must be enforced by your application code before queries execute. Do not use this prompt for single-agent RAG systems, for agents that retrieve from completely disjoint corpora (where overlap is impossible), or when retrieval scopes are naturally partitioned by user-facing filters that agents cannot violate (e.g., hard tenant isolation in a multi-tenant system). Also avoid this prompt when retrieval latency is not a concern and the cost of a downstream deduplication merge is lower than the added coordination step.

Before implementing this prompt, verify that your retrieval system supports the filter or partition mechanisms required to enforce the assigned scopes. If your vector database cannot apply metadata filters at query time, the scope assignments will be advisory only and won't prevent actual overlap. Start by measuring your current retrieval overlap rate—log the document IDs fetched by each agent and compute the Jaccard similarity between their result sets. If overlap is below 10% or your total retrieval volume is small, the coordination overhead may not justify the prompt. If overlap exceeds 20% and retrieval costs are material, proceed to the prompt template and implementation harness sections to wire this into your orchestration layer. The next section provides the copy-ready prompt template with placeholders for your agent intents, index snapshot, and output schema.

This prompt sits in your orchestrator's coordination step, running before any retrieval agents are dispatched. It takes the user's query, the available agent roster with their retrieval capabilities, and any active retrieval scopes already claimed by other agents. The prompt produces a structured assignment that defines exactly which document sources, time windows, or semantic subspaces each agent should retrieve from, ensuring zero overlap. Use this when you have parallel retrieval agents that could independently hit the same vector stores, search APIs, or document collections.

textCopy
You are a retrieval coordination controller in a multi-agent RAG pipeline. Your job is to assign non-overlapping retrieval scopes to agents before they execute any retrieval calls.

## AVAILABLE RETRIEVAL AGENTS
[AGENT_ROSTER]

## ACTIVE RETRIEVAL SCOPES (already claimed by other agents)
[ACTIVE_SCOPES]

## USER QUERY
[USER_QUERY]

## RETRIEVAL CONTEXT
- Total document collections available: [COLLECTION_LIST]
- Time boundaries for this query: [TIME_RANGE]
- Maximum documents per agent: [MAX_DOCS_PER_AGENT]
- Overlap tolerance: zero (no document should be retrieved by more than one agent)

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "assignments": [
    {
      "agent_id": "string",
      "scope": {
        "collections": ["string"],
        "filters": {
          "time_range": {"start": "ISO8601", "end": "ISO8601"},
          "metadata_constraints": {"key": "value"},
          "semantic_bounds": "string describing the semantic subspace to retrieve from"
        },
        "excluded_document_ids": ["string"],
        "max_results": "integer"
      },
      "rationale": "string explaining why this scope was assigned to this agent"
    }
  ],
  "unassigned_coverage": [
    "string describing any part of the query that no agent can cover"
  ],
  "overlap_checks": [
    {
      "agent_pair": ["agent_id_a", "agent_id_b"],
      "potential_overlap": "string describing any boundary risk",
      "mitigation": "string describing how the scopes avoid overlap"
    }
  ]
}

## CONSTRAINTS
1. Every assigned scope must be mutually exclusive. No two agents may retrieve from overlapping collections, time windows, or semantic subspaces.
2. If the query requires coverage that no available agent can provide, list it in `unassigned_coverage` rather than assigning it poorly.
3. Prefer agents with specialized retrieval capabilities matching the scope (e.g., use a legal-document agent for contract collections, not a general web-search agent).
4. If [ACTIVE_SCOPES] already covers part of the query, do not reassign that coverage. Note it as already handled.
5. Every scope must include explicit `excluded_document_ids` if prior retrievals have already fetched specific documents.
6. If the overlap risk between two scopes is non-trivial, include an `overlap_checks` entry explaining the boundary and how it's enforced.

## RISK LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", add a `requires_human_review` boolean field to each assignment and default it to true for any scope touching regulated or customer-specific data.

Adapt this template by replacing the placeholders with your actual agent roster, active scope registry, and collection metadata. The [AGENT_ROSTER] should be a structured list of agent IDs with their retrieval capabilities and specializations. The [ACTIVE_SCOPES] should come from your workstream registry or claim store, populated by a prior step that records what other agents are already retrieving. Wire the output into your dispatch layer: each assignment becomes a task payload for the corresponding agent, and the overlap_checks array feeds your monitoring dashboard for boundary-risk observability. If [RISK_LEVEL] is high, route any assignment with requires_human_review: true to an approval queue before retrieval executes. Avoid using this prompt when only a single retrieval agent exists or when the query is narrow enough that overlap is impossible—the coordination overhead isn't worth it.

This prompt is not a standalone chatbot. It is a control-plane instruction designed to sit inside a multi-agent RAG orchestrator, typically between the planning agent and the retrieval dispatch layer. The orchestrator should invoke this prompt after a user query is decomposed into sub-queries but before those sub-queries are assigned to retrieval agents. The prompt's job is to analyze the set of pending retrieval intents, compare them against an active retrieval registry, and return a structured overlap report that the orchestrator uses to merge, drop, or re-scope retrieval tasks. The implementation harness must treat this prompt as a deterministic gate: no retrieval agent should be spawned until the overlap check passes and the orchestrator has applied the deduplication instructions.

To wire this in, build a pre-retrieval hook in your orchestrator that calls the LLM with the prompt template, passing [PENDING_RETRIEVAL_INTENTS] as a JSON array of objects with intent_id, query, scope_filters, and requesting_agent fields, and [ACTIVE_RETRIEVAL_REGISTRY] as a JSON array of in-flight or recently completed retrieval records with intent_id, query, scope_filters, status, and documents_retrieved. The model must return a strict JSON object matching [OUTPUT_SCHEMA]: an overlap_analysis array where each entry maps a pending intent to one or more active intents with an overlap_score (0-1), a recommendation enum of proceed, merge, drop, or rescope, and a rationale string. Implement a post-generation validator that rejects any output where overlap_score > 0.7 lacks a merge or drop recommendation, or where a rescope recommendation is missing a suggested_scope_filters field. If validation fails, retry once with the validation error injected into [CONSTRAINTS]; if it fails again, escalate to a human operator and block retrieval dispatch. Log every overlap decision with the intent_id, recommendation, and overlap_score for later audit and retrieval overlap rate calculation.

Model choice matters here. This prompt requires strong instruction-following and structured output discipline, not creative reasoning. Use a model with native JSON mode or constrained decoding (e.g., gpt-4o with response_format, Claude 3.5 Sonnet with tool-use-as-output, or a fine-tuned open-weight model with grammar constraints). Set temperature=0 to minimize variance in overlap scoring. The orchestrator should maintain the active retrieval registry in a low-latency store (Redis, DynamoDB) with a TTL on completed entries to prevent unbounded growth. Before deploying, run an eval suite that injects known duplicate intents and measures the overlap detection recall (target >0.95), false-positive rate (target <0.10), and end-to-end latency from prompt invocation to validated output (target <2 seconds). The most common production failure mode is stale registry entries causing false-positive overlap flags; mitigate this by having retrieval agents update their registry status to completed with a timestamp immediately upon finishing, and by excluding completed entries older than [REGISTRY_TTL_SECONDS] from the active registry passed to the prompt.