Backward Compatibility

When a new schema adds a field, older data lacks that field entirely. The reader must substitute a default value to fill the gap. This is the core mechanism enabling backward compatibility.

• Avro: Defaults are declared in the writer's schema and applied by the reader
• Protocol Buffers: Numeric defaults (0, empty string, false) are implicit for new fields
• JSON Schema: Defaults are optional annotations; enforcement depends on the validator

Without defaults, a reader encountering a missing required field would throw a fatal deserialization error.

Backward compatibility is preserved when schema changes are strictly additive. Adding a new optional field or a new enumeration value does not break existing readers processing old data.

• Safe operations: Adding optional fields, adding enum values, adding new types
• Breaking operations: Removing fields, renaming fields, changing field data types
• Semi-safe: Widening a numeric type (int32 to int64) is safe in Protobuf but not in all formats

This constraint forces schema designers to plan for evolution from the start, never removing fields but marking them as deprecated.

In systems like Apache Avro, the reader's schema is the authority. When reading data, the runtime compares the writer's schema (stored with the data) against the reader's expected schema and resolves differences transparently.

• The reader knows what it expects; the writer's schema provides what was written
• Mismatched fields are resolved via schema resolution rules: matching by name, applying defaults, and ignoring unknown fields
• This decouples producers and consumers entirely—neither needs to know the other's version

This pattern is the foundation of schema registry architectures in event-driven systems.

A robust backward-compatible reader must preserve unknown fields when forwarding data. If a consumer reads data with fields it doesn't recognize (added by a newer writer), it must not silently drop them.

• Protocol Buffers: Unknown fields are stored in a reserved section and re-serialized on output
• JSON: Unknown properties are typically discarded unless explicit passthrough logic is implemented
• This enables multi-hop forwarding where intermediate services don't need to understand every field version

Without preservation, data loss cascades through a pipeline as each hop strips unrecognized fields.

Backward-compatible changes map directly to Semantic Versioning (SemVer) rules. A schema change that is backward-compatible warrants a MINOR or PATCH version bump, never a MAJOR.

• MAJOR (X.0.0): Breaking changes—field removal, type change, renaming
• MINOR (0.X.0): Backward-compatible additions—new optional fields, new enum values
• PATCH (0.0.X): Clarifications—documentation fixes, constraint tightening that doesn't affect wire format

Schema registries often enforce compatibility checks against SemVer rules, rejecting any change that would break existing consumers.

Backward compatibility is often confused with full compatibility. A schema change is backward-compatible if new readers can process old data. It is forward-compatible if old readers can process new data. Full compatibility requires both.

• Backward only: Upgrade consumers first, then producers
• Forward only: Upgrade producers first, then consumers
• Full (both): Upgrade in any order; the system tolerates version skew in both directions

In practice, full compatibility is the gold standard for zero-downtime deployments in continuous delivery pipelines.

A centralized service that stores and manages schemas for distributed messaging systems. Confluent Schema Registry is the most widely deployed implementation, providing:

• Versioned schema storage for Avro, Protobuf, and JSON Schema
• Compatibility checking at registration time to prevent breaking changes
• Schema ID embedding in serialized payloads for efficient deserialization Without a registry, producers and consumers must coordinate schema changes out-of-band, creating tight coupling and operational risk.