Partial-Failure Recovery Classification Prompt

This prompt is for data platform engineers and MLOps teams who operate batch processing pipelines where some records succeed and others fail. Its job is to classify failed records by error type and determine the correct recovery strategy—retry, skip, or quarantine—producing a structured remediation plan. Use it when your pipeline produces a partial-failure output and you need a deterministic, auditable decision before the next retry attempt, rather than blindly retrying everything or discarding failures without analysis.

The ideal user has access to the failed record payload, the error message or exception type, the retry attempt count, and any relevant pipeline state (such as whether downstream systems are healthy). The prompt expects these as structured inputs and returns a classification label, a confidence score, a recommended action, and a rationale. It is designed for integration into a retry harness that enforces idempotency, tracks attempt counts, and prevents infinite retry loops. Do not use this prompt for real-time streaming failures where latency constraints preclude classification, or for failures where the root cause is unknown and requires human investigation before any automated recovery decision.

Before deploying, wire the prompt into a pipeline that validates the output schema, enforces max retry limits, and logs every classification decision for audit. For high-risk pipelines—such as financial transaction processing or healthcare data ingestion—require human review for any record classified as quarantine or any record exceeding a configurable retry threshold. The prompt is a classification and routing tool, not a root-cause analysis engine; it tells you what to do with a failure, not why the failure occurred in depth. Pair it with separate observability and debugging workflows for deep failure analysis.

This prompt template is designed to be dropped into a batch processing pipeline's error-handling path. When a batch job completes with partial failures, the system should collect the failed records, their error metadata, and the current retry state, then pass them to this prompt. The model's job is not to fix the data but to classify each failure into a recovery strategy—retry, skip, or quarantine—and produce a machine-readable remediation plan that downstream orchestration logic can execute. The template uses square-bracket placeholders for all dynamic inputs so you can wire it directly into your job runner, Airflow DAG, or stream processor without modifying the instruction structure.

textCopy
You are a batch processing recovery classifier. Your task is to analyze a set of failed records from a data pipeline and produce a structured remediation plan.

## INPUT
You will receive:
- A list of failed records, each with an error type, error message, record identifier, and the current retry attempt count.
- The pipeline's retry policy, including max attempts and backoff strategy.
- The pipeline's idempotency configuration (idempotency key field, deduplication window).

Failed Records:
[FAILED_RECORDS]

Retry Policy:
[RETRY_POLICY]

Idempotency Configuration:
[IDEMPOTENCY_CONFIG]

## CLASSIFICATION RULES
For each failed record, classify it into exactly one recovery strategy:

- **RETRY**: The error is transient (e.g., network timeout, rate limit, temporary downstream unavailability) AND the current attempt count is below the max retry limit. Include a recommended backoff duration in seconds.
- **SKIP**: The error is permanent and non-critical (e.g., single-record schema mismatch, known benign data issue, record already exists and idempotency key confirms successful prior processing). The record can be safely discarded without data loss.
- **QUARANTINE**: The error is permanent and requires investigation (e.g., data corruption, unexpected null in required field, foreign key violation, poison pill pattern). The record must be preserved for manual review.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "remediation_plan": [
    {
      "record_id": "string",
      "error_type": "string",
      "error_message": "string",
      "current_attempt": number,
      "recovery_strategy": "RETRY" | "SKIP" | "QUARANTINE",
      "confidence": number (0.0 to 1.0),
      "rationale": "string (one sentence explaining the classification)",
      "retry_backoff_seconds": number | null,
      "quarantine_reason": "string | null"
    }
  ],
  "summary": {
    "total_failed": number,
    "retry_count": number,
    "skip_count": number,
    "quarantine_count": number,
    "requires_human_review": boolean
  }
}

## CONSTRAINTS
- Do not invent recovery strategies outside RETRY, SKIP, or QUARANTINE.
- If current_attempt equals or exceeds max_attempts, the record must be QUARANTINE or SKIP, never RETRY.
- If an idempotency key is present and matches a previously successful record, prefer SKIP with rationale citing idempotency.
- Set confidence below 0.7 if the error message is ambiguous or the failure pattern is mixed (e.g., timeout that has occurred on every attempt might indicate a persistent downstream issue rather than a transient one).
- For QUARANTINE records, populate quarantine_reason with a specific, actionable description of what needs investigation.
- Return ONLY the JSON object. No markdown fences, no commentary.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

Adaptation guidance: Replace [FAILED_RECORDS] with a JSON array of failure objects from your batch processor's error log. [RETRY_POLICY] should contain your pipeline's max attempts and backoff configuration as a structured object. [IDEMPOTENCY_CONFIG] should specify the field used for deduplication and the time window. The [FEW_SHOT_EXAMPLES] placeholder is critical for production—include 3-5 examples covering each recovery strategy, especially edge cases like a timeout that has recurred on every attempt (should be QUARANTINE, not RETRY) and a schema mismatch on a non-critical optional field (should be SKIP). If your pipeline uses a dead letter queue, wire the QUARANTINE decisions directly to that queue's producer. For high-throughput pipelines, consider batching failed records into groups of 50-100 to stay within context windows while maintaining classification quality.

Before deploying: Run this prompt against a golden dataset of 50-100 known failures with pre-labeled recovery strategies. Measure precision and recall per strategy—RETRY misclassification causes unnecessary compute waste, SKIP misclassification causes data loss, and QUARANTINE misclassification floods your review queue. Set a minimum confidence threshold (we recommend 0.8) below which records should default to QUARANTINE with a human-review flag. Wire the requires_human_review boolean in the summary to your on-call alerting if true. For idempotency-aware pipelines, add an eval that verifies the model never marks a record as RETRY when the idempotency key confirms prior success.

This prompt is designed to be invoked as a post-processing step within a batch pipeline's error-handling path. When a batch processor encounters a set of failed records, it should package the error context—including the original payload, the error message, the stack trace or error code, and the current retry attempt number—into the prompt's [FAILED_RECORDS] and [ERROR_CONTEXT] placeholders. The model's structured output, which classifies each failure as retryable, skip, or quarantine, then becomes the direct input to the pipeline's recovery orchestrator. Do not use this prompt for real-time, single-record failures where latency is critical; it is optimized for bulk triage of accumulated dead letters or batch-level error logs.

To integrate this into an application, wrap the model call in a function that enforces a strict schema validation layer. The expected output is a JSON array of objects, each with a record_id, error_type, recovery_action, and retry_strategy. Before the orchestrator acts on any decision, validate that every record_id from the input batch is present in the output and that no recovery_action is null. Implement a state-tracking mechanism, such as a Redis hash or a database table keyed by record_id, to store the retry count and last error signature. Before executing a retryable action, check this state store to enforce a maximum retry limit and to detect poison pills—records that fail identically on consecutive attempts. For quarantine actions, publish the record and its classification to a dedicated dead-letter queue for manual inspection. Log every classification decision with the model's confidence_score and the prompt version to create an audit trail for debugging classification drift.

Idempotency is the most critical harness requirement. The recovery orchestrator must be able to re-process the same batch of failures without duplicating successful retries or re-quarantining already-inspected records. Achieve this by having the orchestrator acquire a distributed lock per batch before processing and by writing the final status of each record (retried, skipped, quarantined) to the state store atomically. When choosing a model, prefer one with strong JSON mode and low variance at low temperatures (e.g., temperature=0). Start your evaluation suite by testing against a golden dataset of known transient failures (network timeouts), permanent failures (schema violations), and ambiguous cases (timeouts that have already been retried multiple times). The primary failure mode to monitor in production is the model classifying a permanent failure as retryable, which creates an infinite loop. Mitigate this with the hard retry limit in the state store and an alert that fires if any single record_id cycles through the recovery path more than a configured threshold.