Distributed Queue Poison Message Recovery Prompt

The raw body of the poison message that repeatedly failed processing.

{"orderId": "ORD-9921", "amount": -50.00, "currency": "USD"}

Must be valid JSON or string. Null not allowed. Schema check against expected message contract before injection.

Ordered list of failure records including timestamps, error codes, and stack traces from each retry attempt.

[{"attempt": 1, "error": "InvalidAmountError", "timestamp": "2025-03-15T10:00:00Z"}, {"attempt": 2, "error": "InvalidAmountError", "timestamp": "2025-03-15T10:01:00Z"}]

Must be a non-empty array. Each record requires attempt number, error code, and timestamp. Parse check for valid ISO 8601 timestamps.

The name of the queue or topic where the poison message was detected.

order-processing-queue.fifo

Must be a non-empty string matching queue naming conventions. Regex validation against allowed queue name patterns.

The maximum number of retries configured for this queue before dead-lettering.

5

Must be a positive integer. Compare against length of [FAILURE_HISTORY] to detect infinite-loop risk. Retry budget exceeded if failure count >= max retry count.

The dead-letter queue name where unrecoverable messages should be routed.

order-processing-dlq.fifo

Must be a non-empty string. Validate that DLQ exists and is writable before attempting dead-letter action. Null allowed if no DLQ configured, but triggers escalation path.

Identifier for the service or handler that processes messages from this queue.

order-validation-service-v2

Must be a non-empty string. Used to correlate with deployment status and known bug databases. Validate service name against service registry.

The notification channel or endpoint for human escalation when automated recovery fails.

slack://#order-ops-oncall

Must be a valid URI or channel identifier. Parse check for supported schemes (slack, pagerduty, email, webhook). Null triggers default escalation policy.

The number of poison messages within a time window that triggers batch-poisoning detection.

10

Must be a positive integer. If poison count in window exceeds threshold, prompt must recommend batch investigation instead of per-message recovery. Validate against recent queue metrics.

enum: dead_letter | repair_retry | escalate

Must be exactly one of the three allowed values. Reject any other string.

object

true when decision is repair_retry, else null

If repair_retry, must contain action and payload fields. If dead_letter or escalate, must be null.

string

true when repair_instructions is present

Must be a non-empty string describing the specific repair operation. Reject empty or whitespace-only strings.

object

true when repair_instructions is present

Must be a valid JSON object containing the repaired message body. Schema must match the original message schema for the queue.

string

true when decision is escalate

Must be a non-empty string explaining why automated recovery is not possible. Must not exceed 500 characters.

string

true when decision is dead_letter

Must be a non-empty string explaining why the message is unrecoverable. Must not exceed 500 characters.

integer

If present, must be an integer between 1 and [MAX_RETRY_LIMIT]. If absent, the harness default applies.

number

If present, must be a float between 0.0 and 1.0. Values below [CONFIDENCE_THRESHOLD] should trigger human review before executing the decision.

Output contains exactly one of three valid dispositions: dead-letter, repair-and-retry, or escalate. Disposition matches the failure pattern described in [FAILURE_HISTORY].

Output proposes an invalid disposition, proposes multiple conflicting dispositions, or selects a disposition that contradicts documented failure patterns (e.g., dead-letter for a transient network timeout).

Schema check: parse output field disposition and assert value is in enum {dead-letter, repair-and-retry, escalate}. Run 10 golden cases with known failure patterns and assert disposition matches expected label.

When disposition is repair-and-retry, output includes a concrete, executable repair instruction in [REPAIR_INSTRUCTION] that references specific fields from [MESSAGE_PAYLOAD] and corrects the root cause identified in [FAILURE_HISTORY].

Repair instruction is missing, is a generic restatement of the error without a corrective action, or references fields not present in the message payload.

Parse [REPAIR_INSTRUCTION] field. Assert it is non-null when disposition is repair-and-retry. Validate that all referenced field paths exist in the input [MESSAGE_PAYLOAD] schema. Run repair instruction through a dry-run validator that checks for actionable verbs (e.g., transform, replace, set).

Output includes a [RETRY_COUNT_CHECK] field that evaluates whether the message has exceeded [MAX_RETRY_THRESHOLD]. If retry count exceeds threshold, disposition must be dead-letter or escalate, never repair-and-retry.

Output recommends repair-and-retry when [RETRY_COUNT] in [FAILURE_HISTORY] exceeds [MAX_RETRY_THRESHOLD], or [RETRY_COUNT_CHECK] field is missing or evaluates to an incorrect boolean.

Assert that when input [FAILURE_HISTORY].retry_count > [MAX_RETRY_THRESHOLD], output disposition is not repair-and-retry. Assert [RETRY_COUNT_CHECK] field is present and is a boolean matching the threshold comparison.

Output includes a [BATCH_POISONING_RISK] assessment. If the failure pattern indicates a systemic issue (e.g., schema mismatch, upstream data corruption), the assessment flags high risk and disposition must be dead-letter or escalate to prevent blocking the entire queue.

Output recommends repair-and-retry for a systemic failure that would affect all messages in the batch, or [BATCH_POISONING_RISK] assessment is missing or incorrectly flags low risk for a systemic pattern.

Inject test cases with systemic failure signatures (e.g., all messages failing with same schema error). Assert [BATCH_POISONING_RISK] is high and disposition is not repair-and-retry. Assert the assessment references the systemic pattern from [FAILURE_HISTORY].

When disposition is escalate, output includes an [ESCALATION_CONTEXT] object with a concise failure summary, the original [MESSAGE_ID], the [FAILURE_HISTORY] summary, and a recommended [ESCALATION_TARGET] (e.g., on-call, data-engineering, upstream-owner).

Escalation context is missing, omits the message ID, contains an overly verbose or unactionable summary, or fails to recommend a specific escalation target.

Parse [ESCALATION_CONTEXT] object. Assert required fields are present: summary, message_id, failure_summary, escalation_target. Assert summary length is under 500 characters. Validate escalation_target is a non-empty string.

When disposition is dead-letter, output includes a [DEAD_LETTER_METADATA] object containing the original [MESSAGE_ID], [FAILURE_HISTORY], the reason for dead-lettering, and a timestamp. This ensures the message can be audited or reprocessed later.

Dead-letter metadata is missing, omits the failure history, or lacks a clear reason string explaining why the message was dead-lettered rather than repaired or escalated.

Parse [DEAD_LETTER_METADATA] object. Assert required fields: message_id, failure_history, dead_letter_reason, dead_lettered_at. Assert dead_letter_reason is a non-empty string that references specific failures from [FAILURE_HISTORY].

Output is valid JSON matching the expected schema: top-level fields include disposition, repair_instruction, retry_count_check, batch_poisoning_risk, escalation_context, dead_letter_metadata. All required fields for the chosen disposition are present and non-null.

Output is not valid JSON, is missing top-level fields, contains null values for fields required by the chosen disposition, or includes extra fields that violate the output contract.

Validate output against JSON Schema. Assert parse succeeds. For each disposition value, assert that the corresponding required fields are present and non-null. Assert no unexpected top-level fields are present.

Every claim in the output (repair instruction, escalation summary, dead-letter reason) is grounded in specific entries from [FAILURE_HISTORY]. The output does not invent failure modes not present in the input.

Output references a failure mode, error code, or stack trace not present in [FAILURE_HISTORY], or makes a generic recommendation without citing specific failure entries.

Extract all error codes, exception types, and failure descriptions from output. Assert each is a substring match or semantic match to content in [FAILURE_HISTORY]. Flag any novel failure descriptions as hallucination.