Kafka Message Deserialization Error Recovery Prompt

This prompt is for streaming data engineers and platform operators who need to recover a consumer that has failed to deserialize a message from a Kafka topic. The job-to-be-done is to take a raw, hex-encoded byte payload, the expected schema, and the specific deserialization error, and produce either a best-effort parsed record or a structured dead-letter entry with a poison-pill flag. The ideal user is someone on-call or automating a dead-letter queue (DLQ) handler who needs a programmatic, repeatable recovery step—not a one-off debugging session. Required context includes the raw message bytes, the exact error message from the deserializer, and the target schema (Avro, Protobuf, JSON Schema, or similar).

Do not use this prompt when the root cause is a systemic producer bug that requires a code fix, when the schema registry is unavailable and the schema itself is unknown, or when the message payload is encrypted and the decryption keys are missing. In those cases, the prompt cannot produce a safe recovery and will likely hallucinate a plausible but incorrect record. This prompt is also not a substitute for a schema evolution compatibility layer; it is a tactical recovery tool for individual malformed messages that have already landed in a topic. Use it inside a retry harness that respects a retry budget and escalates after repeated failures.

Before wiring this into a production DLQ handler, test it against a golden dataset of known deserialization failures with expected recovery outputs. Evaluate the prompt on three axes: schema compliance (does the output match the target schema?), semantic preservation (did the recovery drop or corrupt critical fields?), and poison-pill flag accuracy (did it correctly identify unrecoverable messages?). The next section provides the copy-ready template. After adapting it, review the implementation harness section to understand validation, logging, and escalation requirements.

This prompt template is designed to be copied directly into your recovery harness. It instructs the model to act as a streaming data reliability engineer, diagnosing a Kafka consumer record that failed deserialization. The model receives the raw bytes (hex-encoded for safe transport), the expected schema, and the exact deserialization error. Its job is to produce a best-effort parsed record or a structured dead-letter entry with a poison-pill flag. Use this template as the core instruction block in a retry loop, passing in the failure context from your consumer's error handler.

textCopy
You are a streaming data reliability engineer diagnosing a Kafka message deserialization failure.

Your task is to analyze the raw bytes, expected schema, and error message, then produce one of two outputs:
1. A best-effort parsed record that conforms as closely as possible to the expected schema.
2. A dead-letter entry with a poison-pill flag if recovery is impossible.

## INPUT
- Raw Bytes (hex-encoded): [RAW_BYTES_HEX]
- Expected Schema: [EXPECTED_SCHEMA]
- Deserialization Error: [ERROR_MESSAGE]
- Topic: [TOPIC_NAME]
- Partition: [PARTITION]
- Offset: [OFFSET]

## CONSTRAINTS
- Never invent data. If a field cannot be recovered, mark it as null and add an entry to the `recovery_notes` map explaining why.
- If the raw bytes are completely unparseable or the schema mismatch is irrecoverable, set `poison_pill` to true and populate `dead_letter_reason`.
- Preserve the original offset, partition, and a hex prefix of the raw bytes in the output for traceability.
- If you produce a best-effort record, include a `confidence` score between 0.0 and 1.0 for the overall recovery.

## OUTPUT_SCHEMA
Return a single JSON object with this exact structure:
{
  "poison_pill": boolean,
  "dead_letter_reason": string | null,
  "recovered_record": object | null,
  "recovery_notes": object,
  "confidence": number,
  "original_offset": number,
  "original_partition": number,
  "raw_bytes_prefix": string
}

## EXAMPLES
Example 1: Recoverable field-level corruption
Input Error: "Unexpected character at position 42 while parsing field 'timestamp'"
Output: poison_pill=false, recovered_record contains all fields except 'timestamp' which is null, recovery_notes={"timestamp": "Unparseable, set to null"}, confidence=0.85

Example 2: Complete garbage bytes
Input Error: "Unknown magic byte 0xFF at start of record"
Output: poison_pill=true, dead_letter_reason="Unrecognized serialization format. Magic byte 0xFF does not match Avro, Protobuf, or JSON.", recovered_record=null, confidence=0.0

## RISK_LEVEL
Medium. Incorrect recovery can corrupt downstream aggregates. Always log recovery decisions and route poison-pill messages to a dead-letter topic for human review.

To adapt this template, replace the square-bracket placeholders with values from your consumer's exception handler. The RAW_BYTES_HEX field expects a hexadecimal string representation of the raw Kafka message bytes—use bytes.hex() in Python or equivalent in your language. The EXPECTED_SCHEMA should be the full schema definition (Avro, Protobuf, or JSON Schema) as a string. If your schema is large, consider including only the relevant record definition to stay within context limits. Wire the output into a validator that checks the JSON structure before routing the recovered record back to the happy path or the poison pill to the dead-letter topic. Always increment a retry counter and enforce a retry budget—if this prompt fails to produce valid JSON after two attempts, escalate the raw bytes to a human-operable dead-letter queue with the full error context.

This prompt is designed to sit inside a Kafka consumer's exception handler, invoked when poll() returns a record that fails deserialization. The consumer should catch the SerializationException (or equivalent for your client library), extract the raw bytes and the expected schema, and call the LLM before committing offsets. The prompt is not a replacement for schema registry validation or poison-pill handling—it is a best-effort recovery layer that attempts to salvage data that would otherwise be discarded. Wire it as a synchronous call within the consumer poll loop, but protect the consumer's liveness with a strict timeout (e.g., 2 seconds) and a circuit breaker that disables LLM recovery if the model endpoint returns 5xx errors or exceeds latency thresholds for more than 10 consecutive attempts.

The harness must enforce several constraints before and after the LLM call. Pre-call: hex-encode the raw bytes to prevent tokenization artifacts, truncate the payload to a safe upper bound (e.g., 8 KB of hex), and attach the exact deserialization error message and stack trace. Include the expected schema as a JSON Schema or Avro schema string, not a schema registry ID—the model needs the structural definition, not a pointer. Post-call: validate that the returned payload parses successfully against the expected schema using your existing deserializer. If it parses, log the repair event with the original offset, partition, and a hash of the raw bytes for audit, then process the record normally. If it fails to parse, or if the model returns a poison_pill flag, route the original bytes and error context to a dead-letter topic with headers that include recovery_attempted: true, model_id, and failure_reason. Never retry the LLM call more than once for the same record—the prompt already includes a best-effort instruction, and a second attempt rarely succeeds when the first failed.

For production deployment, choose a fast, inexpensive model for this recovery path (e.g., GPT-4o-mini, Claude Haiku, or a fine-tuned small model if your schema surface is narrow). The prompt is structured to produce a compact JSON response, so set max_tokens to a value proportional to your expected record size (e.g., 512–1024 tokens). Instrument the harness with metrics: deserialization_failures_total, llm_recovery_attempts_total, llm_recovery_successes_total, and dead_letter_routed_total. These metrics feed your retry budget and escalation logic—if the recovery success rate drops below 50% over a rolling window, the harness should alert and consider disabling LLM recovery until the root cause (e.g., a producer schema bug) is fixed. The prompt is a tactical recovery tool, not a substitute for schema governance.