Throughput-Aware Batching Prompt for Rate-Limited Pipelines

This prompt is for infrastructure and data platform engineers who must feed a high volume of heterogeneous items into a downstream system governed by strict rate limits. The core job-to-be-done is not simple round-robin distribution or fixed-size batching; it is to produce a batch assignment plan that maximizes total throughput while respecting constraints like requests-per-second caps, token-per-minute budgets, or concurrent connection ceilings. The prompt expects a pre-classified list of items, each annotated with a priority tier (e.g., critical, high, standard, low), a payload size estimate (in tokens, bytes, or records), and an estimated processing cost. It reasons across these dimensions to group items into batches that are sized to stay under limits, ordered to prevent high-priority starvation, and structured to allow downstream workers to process them without throttling errors. Use this when your pipeline has variable item sizes, mixed business urgency, and rate limits that make naive batching either wasteful or violation-prone.

The ideal user is an engineer who already has a working ingestion or dispatch pipeline and needs to replace a static batching function with one that adapts to payload characteristics and priority signals. You must provide the prompt with structured input: a JSON array of items where each object includes item_id, priority (one of the defined tiers), size_estimate (in your pipeline's relevant unit), and cost_estimate (in the downstream system's limiting unit, such as tokens). The prompt also requires a [CONSTRAINTS] block specifying the rate limits, such as max_requests_per_second, max_tokens_per_minute, and max_concurrent_connections. Without these concrete constraints, the model cannot produce a valid batching plan. The output is a batch assignment plan: a list of batches, each with a batch_id, items (array of item IDs), estimated_size, estimated_cost, recommended_processing_window_start, and starvation_risk flags for any priority tier that has been waiting beyond its acceptable latency threshold. The prompt also produces fairness metrics showing the distribution of batch slots across priority tiers.

Do not use this prompt for real-time streaming dispatch with sub-second latency requirements. A deterministic scheduler with fixed rules will be faster and more predictable for that use case. Do not use it when all items have uniform size and equal priority; a simple token-bucket or leaky-bucket algorithm is sufficient there. Do not use it as a substitute for downstream rate limiting enforcement—this prompt produces a plan, but your application harness must still enforce the limits and handle cases where the downstream system rejects a batch despite the plan. The prompt is also not a real-time starvation monitor; it assesses starvation risk at planning time based on the current queue snapshot, but it cannot react to new items arriving mid-execution. For that, you need a streaming scheduler that re-plans continuously. Finally, avoid this prompt when the downstream system's rate limits are unknown or unstable; the model needs concrete numbers in [CONSTRAINTS] to produce a valid plan, and guessing will lead to throttling or underutilization.

This prompt template is designed to be wired into a batch processing pipeline where a downstream system enforces strict rate limits. It accepts a list of items, each with a payload size, priority tier, and arrival timestamp, and produces a batch assignment plan. The model's job is to group items into optimally sized batches that respect the downstream system's requests-per-second (RPS) and payload-per-request limits while preventing starvation of lower-priority items. The template uses square-bracket placeholders for all dynamic inputs, making it safe to populate from your pipeline's runtime state before each inference call.

textCopy
You are a throughput-aware batch planner for a rate-limited downstream system. Your task is to classify and group incoming items into optimally sized batches.

## DOWNSTREAM CONSTRAINTS
- Maximum requests per second: [MAX_RPS]
- Maximum payload size per request: [MAX_PAYLOAD_SIZE]
- Current processing window start: [WINDOW_START_TIMESTAMP]
- Current processing window end: [WINDOW_END_TIMESTAMP]

## PRIORITY TIER DEFINITIONS
[PRIORITY_TIER_DEFINITIONS]

## FAIRNESS POLICY
- No priority tier may be starved for more than [MAX_STARVATION_SECONDS] seconds.
- Within each second, allocate capacity proportionally: [TIER_CAPACITY_ALLOCATION]
- Items that have waited longer than [MAX_WAIT_SECONDS] must be escalated to the next available batch regardless of tier allocation.

## INPUT ITEMS
Each item is a JSON object with the following fields:
- id: unique identifier
- payload_size: size in bytes
- priority_tier: one of the defined tiers
- arrival_timestamp: ISO 8601 timestamp of when the item entered the queue

[INPUT_ITEMS]

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "batches": [
    {
      "batch_id": "string",
      "items": ["item_id_1", "item_id_2"],
      "total_payload_size": number,
      "estimated_processing_window_start": "ISO 8601 timestamp",
      "estimated_processing_window_end": "ISO 8601 timestamp",
      "priority_tier_breakdown": {
        "[TIER_NAME]": count
      }
    }
  ],
  "unbatched_items": ["item_id"],
  "starvation_alerts": [
    {
      "tier": "string",
      "item_ids": ["item_id"],
      "wait_duration_seconds": number,
      "reason": "string"
    }
  ],
  "capacity_utilization_pct": number
}

## CONSTRAINTS
- Do not exceed [MAX_PAYLOAD_SIZE] per batch.
- Do not schedule more than [MAX_RPS] batches per second.
- If an item cannot be placed in any batch within the processing window, add it to unbatched_items with a reason.
- If a priority tier has items waiting longer than [MAX_STARVATION_SECONDS], generate a starvation_alert.
- Prefer grouping items of the same priority tier together, but allow mixed-tier batches when necessary to prevent starvation.

To adapt this template, replace each square-bracket placeholder with data from your pipeline's runtime state. The [INPUT_ITEMS] placeholder should be populated with a JSON array of item objects serialized as a string. The [PRIORITY_TIER_DEFINITIONS] placeholder should contain a clear description of each tier and its business meaning. Before deploying, validate that the model's output conforms to the specified schema using a JSON schema validator. If the output fails validation, implement a retry with the validation error message included as additional context. For high-stakes pipelines where incorrect batching could cause SLA violations, route batches flagged with starvation_alerts to a human reviewer before execution.

This prompt is designed to sit between an upstream queue and a rate-limited downstream API. It receives a batch of pending items, each tagged with a priority tier and estimated payload size, and returns a set of batch assignment decisions that respect the downstream system's requests-per-second (RPS) limit, per-batch size cap, and priority ordering rules. The prompt does not execute the batches—it produces a plan that your application code must enact. The harness is responsible for feeding the prompt with accurate rate-limit state, consuming its output, and handling cases where the plan is incomplete or invalid.

Input assembly requires you to collect three data structures before calling the model: (1) the current rate-limit window state, including remaining capacity and reset time; (2) a list of pending items, each with a unique item_id, priority (e.g., critical, high, standard, low), and estimated_tokens or payload_bytes; and (3) the downstream system's constraints, such as max_rps, max_batch_size, and max_payload_bytes_per_batch. Pack these into the [PENDING_ITEMS], [RATE_LIMIT_STATE], and [DOWNSTREAM_CONSTRAINTS] placeholders. The model returns a JSON array of batch assignments, each containing an item_ids list, an estimated_completion_window, and a priority_level. Validation must happen in the harness, not in the prompt. After receiving the output, validate that: every item appears exactly once across all batches; no batch exceeds max_batch_size or max_payload_bytes_per_batch; the total estimated processing time across batches does not exceed the rate-limit window; and higher-priority items are not starved by lower-priority batches. If validation fails, do not retry with the same input—log the failure, flag the violating items, and fall back to a deterministic round-robin or priority-queue algorithm.

Retry and fallback logic should be layered. If the model returns malformed JSON, retry once with a stricter output schema instruction appended to the prompt. If the second attempt fails, route the pending items to a manual review queue or a rule-based batcher. For transient model unavailability, use exponential backoff with a maximum of three retries before falling back. Starvation prevention is a harness-level concern: track how long each priority tier has been waiting and inject a [STARVATION_WINDOW] parameter into the prompt that forces the model to include items that have exceeded their maximum wait time, regardless of throughput optimization. Log every batch assignment decision with the model's rationale, the validated batch plan, and any overrides applied by the harness. This audit trail is essential for debugging priority inversion and rate-limit exhaustion incidents.

Model choice matters here. Use a model with strong JSON output discipline and low latency—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Flash are good candidates. Avoid models that struggle with structured output or introduce high variance in response times, because the batching decision itself must complete within a fraction of the rate-limit window to be useful. If your pipeline processes thousands of items per second, consider running this prompt on a sampled subset of the queue and extrapolating batch assignments, or using a smaller classifier model to pre-group items by priority before calling the full batching prompt. Tool use is not required for this prompt, but if your downstream system exposes a real-time rate-limit status API, wire it into the harness so the [RATE_LIMIT_STATE] placeholder is always populated with live data rather than stale estimates.

What to avoid: Do not use this prompt for pipelines where items have hard real-time deadlines measured in milliseconds—the model call adds latency that may violate those constraints. Do not rely on the prompt to detect duplicate items or handle idempotency; deduplication must happen upstream. Do not skip the harness-level validation step; the model can produce plausible-looking but incorrect batch assignments, especially when the pending item list is large or the rate-limit state is near exhaustion. Finally, do not treat this prompt as a replacement for a proper scheduler when your system requires fairness guarantees that must be provably correct—use it as an optimization layer on top of a deterministic baseline, and always keep the fallback path tested and ready.