Async Tool Call Fan-Out and Scatter-Gather Prompt Template

This prompt is designed for agent developers who need to parallelize a single logical task across multiple asynchronous tool calls and then aggregate the results. The core job-to-be-done is transforming a high-level objective into a structured dispatch plan that your application harness can execute concurrently. Use it when a task can be partitioned into independent shards, such as querying multiple databases with different parameters, calling several APIs to gather data for a comparison, or processing chunks of a large document in parallel to extract entities. The prompt instructs the model to generate a list of tool invocations, interpret the partial results as they return, handle individual shard failures gracefully, and produce a single consolidated output that answers the original objective.

This playbook assumes you have an application harness that can receive the model's structured dispatch plan, execute the specified tool calls asynchronously, and return their results to the model's context for the gather step. The prompt is not a standalone magic function; it is a contract between the model's reasoning and your execution layer. You must provide the model with a precise list of available tools, their schemas, and any constraints like rate limits or timeouts. The model will then output a machine-readable plan, typically a JSON array of tool call definitions. Your harness parses this plan, fans out the calls, collects the responses, and feeds them back into the model with the original prompt for final aggregation. This separation of planning and execution is critical for reliability and testability.

Do not use this prompt for strictly sequential tool chains where each step depends on the previous output, as the fan-out pattern introduces non-deterministic ordering that will break dependency chains. It is also unsuitable for real-time streaming scenarios where incremental partial results must be surfaced to a user immediately, because the gather step requires all shards to complete before consolidation. Avoid this pattern when the cost of an unnecessary fan-out (e.g., spinning up 100 parallel calls for a task that could be done with 2) outweighs the latency benefit, or when the tools have side effects that cannot be safely parallelized. For high-risk operations, ensure your harness includes a human approval gate before executing the fan-out plan, especially when the tool calls involve destructive actions or access regulated data.

This template gives an agent explicit instructions for dispatching parallel async tool calls, tracking in-flight requests, detecting completion, handling partial failures, and reducing results into a coherent final output. It is designed for agent developers who need reliable scatter-gather behavior when tools have variable latency and independent failure modes. Replace the square-bracket placeholders with your specific tool definitions, output schema, and operational constraints before use.

textCopy
You are an async tool-call orchestrator. Your job is to fan out work across multiple independent tool calls, track their completion, handle partial failures, and reduce results into a single structured output.

## TOOLS AVAILABLE
[TOOLS]

## FAN-OUT RULES
1. When given a task that can be parallelized, identify all independent work units.
2. Dispatch all tool calls simultaneously. Do not serialize independent calls.
3. For each dispatched call, record a correlation ID and the expected result shape.
4. If a tool requires input from another tool's output, sequence those calls explicitly and note the dependency.

## GATHER RULES
1. Track completion status for every dispatched call: `pending`, `completed`, `failed`, `timed_out`.
2. Do not proceed to final output until ALL calls have reached a terminal state OR the [GATHER_TIMEOUT] threshold is reached.
3. For calls that fail or time out, classify the failure:
   - `recoverable`: retry with backoff up to [MAX_RETRIES] times.
   - `non_recoverable`: mark as failed and continue with partial results.
   - `degraded`: accept partial or stale result if available.
4. If more than [CRITICAL_FAILURE_THRESHOLD] of calls fail, abort the gather and report the failure state.

## RESULT REDUCTION RULES
1. Merge successful results according to the output schema.
2. For failed calls, include a `partial_failures` block in the output with:
   - `call_id`
   - `failure_reason`
   - `impact_on_result`: what is missing or degraded
3. If a result conflicts with another result, apply the conflict resolution strategy: [CONFLICT_STRATEGY].
4. Do not fabricate data for failed or missing calls.

## OUTPUT SCHEMA
Your final output must conform to this schema:
[OUTPUT_SCHEMA]

## CONSTRAINTS
- Maximum concurrent tool calls: [MAX_CONCURRENCY]
- Gather timeout: [GATHER_TIMEOUT] seconds
- Max retries per call: [MAX_RETRIES]
- Critical failure threshold: [CRITICAL_FAILURE_THRESHOLD]
- Idempotency: generate and attach idempotency keys for all mutating calls

## BEFORE FINALIZING
1. Verify all terminal states are accounted for.
2. Confirm no fabricated data in partial-failure slots.
3. Validate output against the schema.
4. If [HUMAN_REVIEW_REQUIRED] is true for any failure class, pause and request review before returning the final output.

To adapt this template, start by defining your tool set in the [TOOLS] block with clear schemas, argument descriptions, and failure modes. Set [GATHER_TIMEOUT] based on your p99 tool latency plus buffer. Choose [CONFLICT_STRATEGY] from options like last-write-wins, majority-vote, manual-resolution, or fail-on-conflict. For high-risk workflows such as financial transactions or clinical data, set [HUMAN_REVIEW_REQUIRED] to true for any non-recoverable failure class. Wire the output into a validation layer that checks schema compliance and flags missing required fields before the result reaches downstream consumers.

This prompt template is designed to sit at the orchestration layer of an agent application, not as a standalone chat interaction. The application is responsible for providing the list of available async tools, their schemas, the user's objective, and any constraints. The model's output is a structured dispatch plan that the application must parse, validate, and execute. The prompt does not make the tool calls itself; it produces the instructions for the application's execution engine to fan out work, collect results, and decide on a gather strategy. The application must treat the model's output as a plan to be verified, not as executable code to be run blindly.

The implementation harness requires several components working together. First, a schema validator must confirm the model's output matches the expected JSON structure, including the fan_out array of tool call specifications, the gather_strategy object, and the failure_policy block. Each tool call in the fan-out must reference a tool that exists in the application's tool registry, with arguments that conform to that tool's input schema. Second, a dispatch executor must issue the async calls with generated correlation IDs, respecting any concurrency limits and rate limits defined in the application configuration. Third, a result collector must track in-flight calls, handle partial completions, and detect stragglers using configurable timeouts. Fourth, a reduce function must apply the gather strategy—whether it's a simple concatenation, a voting mechanism, a merge with conflict resolution, or a weighted aggregation—to produce the final aggregated result. Each of these components should emit structured logs with trace IDs for observability.

For production reliability, implement a straggler timeout per fan-out batch. If any tool call exceeds the timeout, the gather strategy must proceed with partial results, and the application must log the missing shard with its correlation ID for later reconciliation. Add a result freshness check that compares the completion timestamp of each result against a configurable staleness threshold; stale results should be discarded or flagged. For high-risk domains, insert a human review gate before the final aggregated output is returned to the user or downstream system, especially when the gather strategy involves conflict resolution or when partial failures occurred. The review gate should present the original objective, the individual tool results, the proposed aggregation, and any failure flags. Do not rely on the model to self-report errors; the application harness must independently verify output completeness and consistency.

When wiring this into an existing agent framework, avoid the temptation to let the agent loop freely on fan-out decisions. Instead, treat the prompt as a single-turn planning step: the model produces the dispatch plan, the application executes it, and a separate reduce step produces the final output. If the gather step detects irreconcilable conflicts or missing critical shards, the application can re-invoke the prompt with the partial results and a specific request for conflict resolution, but this should be a deliberate retry with a bounded loop, not an open-ended agent cycle. Log every dispatch, every result, every timeout, and every reduce decision so that production failures can be traced to a specific shard or gather logic error rather than requiring a full replay of the agent's reasoning.