Avro Schema Evolution Repair Prompt

This prompt is for data platform teams and stream-processing engineers who encounter Avro schema compatibility failures at the point of consumption. The job-to-be-done is straightforward: a producer has written records using a writer schema that conflicts with the consumer's reader schema, and you need a programmatic resolution that preserves as much data as possible without manual record-by-record patching. The ideal user is an engineer integrating this prompt into an automated dead-letter queue (DLQ) recovery pipeline or a schema-registry remediation workflow. Required context includes the full writer schema, the full reader schema, and a sample of the failing records that triggered the compatibility error.

Do not use this prompt when the schema conflict is intentional and the correct behavior is to reject the records outright, or when the data loss risk is unacceptable for the domain (e.g., financial transactions where a default fill could misstate a value). This prompt is also inappropriate when the root cause is a producer bug that should be fixed upstream rather than patched at the consumer. The prompt assumes you have already ruled out simple solutions like updating the consumer to a newer schema version or adjusting the compatibility mode in your schema registry. It is a recovery tool for production incidents, not a substitute for schema design governance.

The prompt works best when you have a clear compatibility resolution policy defined before invocation: decide whether you prefer default fills, projection (dropping fields), or union widening for type mismatches. If your organization requires human approval for data modifications, wire this prompt into a review queue rather than an automated repair loop. The output should be treated as a candidate repair that requires validation against the reader schema and a semantic-diff check to confirm no critical fields were silently dropped or corrupted. For high-risk pipelines, always log the original failing record, the repair decision, and the corrected record for auditability.

This template is the core instruction set for an AI model to act as a schema evolution repair engine. It is designed to be copied directly into your prompt management system, IDE, or orchestration harness. The prompt takes three critical inputs—the writer schema, the reader schema, and a sample of failing records—and forces the model to reason about the compatibility break before proposing a fix. Use square-bracket placeholders to inject your specific schemas, records, and operational constraints at runtime. The template is structured to separate the diagnostic reasoning from the repair action, making the output auditable.

codeCopy
You are an Avro schema evolution repair specialist. Your task is to diagnose a compatibility conflict between a writer schema and a reader schema, then repair a batch of failing records so they can be successfully deserialized by the consumer.

## INPUT

[WRITER_SCHEMA]
[READER_SCHEMA]
[FAILING_RECORDS]

## CONSTRAINTS

- [CONSTRAINTS]

## INSTRUCTIONS

1.  **Diagnose the Conflict:** Compare the writer and reader schemas. Identify every field that causes a schema resolution error (type mismatch, missing default, namespace conflict, etc.). Explain the root cause of the incompatibility.
2.  **Propose a Resolution Strategy:** For each conflicting field, choose exactly one resolution action from the list below. Justify your choice based on Avro schema evolution rules and the provided constraints.
    - `default_fill`: Apply a sensible default value for a new field in the reader schema that is missing in the writer records.
    - `projection`: Drop a field from the writer records that is not present in the reader schema.
    - `union_widening`: Wrap a field's value in a union type to match a widened reader schema (e.g., `int` to `["int", "long"]`).
    - `type_coercion`: Perform a safe type conversion (e.g., `int` to `string`) if the constraints allow it.
    - `reject`: Mark the record as irreparable if no safe resolution exists.
3.  **Repair the Records:** Apply your resolution strategy to each record in the [FAILING_RECORDS] list. Produce a corrected version of each record that conforms to the [READER_SCHEMA].
4.  **Flag Unresolvable Issues:** If any record cannot be fully repaired, flag it with a `repair_status` of `"partial"` or `"failed"` and provide a clear reason.

## OUTPUT_SCHEMA

Your response must be a single, valid JSON object conforming to this structure:

{
  "diagnosis": {
    "incompatibility_type": "string",
    "conflicting_fields": [
      {
        "field_name": "string",
        "writer_type": "string",
        "reader_type": "string",
        "resolution_strategy": "default_fill | projection | union_widening | type_coercion | reject",
        "justification": "string"
      }
    ]
  },
  "repaired_records": [
    {
      "original_record": {},
      "repaired_record": {},
      "repair_status": "success | partial | failed",
      "repair_log": ["string"]
    }
  ]
}

[OUTPUT_SCHEMA]

To adapt this template, start by replacing the core placeholders. [WRITER_SCHEMA] and [READER_SCHEMA] should contain the full Avro schema JSON, not just a name or ID. [FAILING_RECORDS] is a JSON array of the problematic records. The [CONSTRAINTS] placeholder is critical for safety; use it to inject business rules like "never drop a field marked as PII" or "prefer default_fill over projection for all non-nullable fields." The [OUTPUT_SCHEMA] placeholder allows you to enforce a strict contract for your downstream parser, such as adding required fields or enum values. If your use case is high-risk, add a final instruction to set a confidence_score for each repair and to flag any record for human review where the score is below a threshold.

This prompt is designed to sit inside a dead-letter queue (DLQ) handler or a schema-registry compatibility check failure path. When a consumer fails to deserialize a record because the writer schema has diverged from the reader schema, the harness should catch the SchemaEvolutionException, extract the writer schema (from the message header or registry), the reader schema (from the consumer's local cache or registry), and a sample of the raw failing bytes. The prompt is then invoked to produce a compatibility resolution action and, if possible, a corrected record. The output is not a final answer but a structured repair instruction that the harness must validate and execute.

The harness must enforce a strict validation layer before any corrected record is forwarded downstream. After the model returns a resolution, the harness should: (1) parse the output against a fixed JSON schema that includes resolution_type (one of default_fill, projection, union_widening, reject), corrected_record, and explanation; (2) if resolution_type is not reject, serialize the corrected_record using the reader schema and confirm it succeeds without error; (3) run a semantic-diff check comparing the original writer-schema fields against the corrected record to ensure no non-nullable required fields were silently dropped without an explicit default_fill annotation; (4) log the model's explanation and the diff result to an audit table. If any validation step fails, the record should be routed to a manual-review quarantine topic, not retried automatically. For high-throughput pipelines, consider caching model responses for identical (writer_schema, reader_schema) pairs to avoid redundant LLM calls.

Model choice matters here. Use a model with strong structured-output capabilities and a context window large enough to hold both schemas plus the failing record sample. For most Avro schemas under 4KB, a model like gpt-4o or claude-3-5-sonnet with JSON mode enabled is sufficient. Set temperature to 0 to maximize deterministic repair behavior. The prompt should be called with a single retry on validation failure; if the second attempt also fails validation, escalate to the quarantine topic. Do not loop beyond two attempts, as schema evolution conflicts that the model cannot resolve in two tries are likely ambiguous and require human judgment. Wire the harness to emit a schema_evolution_resolution metric (tagged with resolution_type and success) so the team can monitor the rate of automated repairs versus quarantines over time.