Schema Evolution

The guarantee that data written with an older schema version can be successfully read and processed by applications using a newer schema version. This is the most critical property for data pipelines where consumers upgrade before producers.

• Mechanism: Achieved by only adding optional fields with default values or removing fields that consumers already ignore.
• Example: Adding a new phone_number field with a default value of null to a User schema is backward compatible because old data without this field can still be deserialized by the new reader.
• Violation: Removing an existing required field or changing its data type breaks backward compatibility, causing runtime deserialization errors.

The guarantee that data written with a newer schema version can be successfully read and processed by applications using an older schema version. This property is essential when producers are updated before consumers.

• Mechanism: Achieved by only removing optional fields or adding fields that older readers can safely ignore using a schema registry that provides the writer's schema to the reader.
• Example: A producer adds a middle_name field to a Customer record. An older consumer that doesn't know about middle_name can still process the record by ignoring the unknown field.
• Violation: Adding a new required field without a default value breaks forward compatibility, as older consumers cannot provide a value for it.

A schema change that is simultaneously backward compatible and forward compatible. Data can flow seamlessly in both directions between old and new schema versions, allowing independent upgrade cycles for all producers and consumers.

• Mechanism: Achieved by only adding optional fields with default values. No fields are ever removed or made required.
• Example: Adding an optional tags array with a default of [] to an Article schema. Old readers ignore the field; new readers handle its absence gracefully.
• Use Case: Critical for event-driven architectures and microservices where services are deployed independently and cannot be upgraded atomically.

A centralized service that stores and validates schemas, acting as the gatekeeper for compatibility. It prevents breaking changes from being committed to production by checking every new schema version against configured compatibility rules.

• Compatibility Types Enforced:
  • BACKWARD: New schema can read old data
  • FORWARD: Old schema can read new data
  • FULL: Both backward and forward
  • NONE: No checks (dangerous)
• Workflow: A producer registers its schema. The registry validates it against the previous version. If the check fails, the producer is blocked from sending incompatible data.

The guarantee that a system using a newer schema can read data written with an older schema. This is the most critical safety net in data pipelines.

• Mechanism: Achieved by only adding optional fields or setting default values for new fields.
• Failure Mode: A consumer crashes because a new required field is missing from old data.
• Real-World Example: A Kafka consumer on version 2.1 must still process events produced by version 2.0 without deserialization errors.

The guarantee that a system using an older schema can read data written with a newer schema. This protects consumers from producers that have been upgraded ahead of them.

• Mechanism: Achieved by never removing fields and using generic container types for unknown data.
• Critical Rule: Schemas must be designed to ignore unknown fields rather than throwing strict parsing errors.
• Protocol Buffers: Handles this natively; unknown fields are simply preserved and ignored during deserialization.

A centralized governance layer that acts as the source of truth for all schemas in a distributed event-driven architecture. It enforces compatibility rules before changes are committed.

• Core Function: Stores a versioned history of every schema (Avro, Protobuf, JSON Schema).
• Compatibility Checks: Rejects producer registration if a new schema violates the configured compatibility mode (BACKWARD, FORWARD, FULL).
• Efficiency: Data is serialized with a short schema ID rather than the full schema text, drastically reducing wire overhead.