Schema Evolution

The most critical safety net. A new schema version is backward compatible if data written with the new schema can be read by consumers using the previous schema version.

• Mechanism: Achieved by only adding optional fields or deleting fields with default values.
• Constraint: You must never delete a required field or change an existing field's type.
• Use Case: Upgrading producers first without breaking any existing downstream dashboards or control systems.
• Example: Adding an optional vibration_threshold field to a sensor reading schema.

A schema is forward compatible if data written with an old schema can be read by consumers using the new schema version. This is harder to guarantee.

• Mechanism: Requires consumers to be tolerant of missing fields, typically by defining default values for all fields in the schema definition.
• Constraint: You cannot add a new required field without a default value.
• Use Case: Upgrading consumers first, or deploying to a fleet where consumer rollout is faster than producer rollout.
• Protocol Support: Formats like Apache Avro and Protocol Buffers natively support this through reader/writer schema resolution.

The gold standard for zero-downtime deployments. A schema change is fully compatible if it is simultaneously both backward and forward compatible.

• Guarantee: Producers and consumers can be upgraded in any order, at any time, without any serialization errors.
• Safe Operations: Only operations like adding an optional field with a default value or removing an optional field meet this strict criterion.
• Industrial Relevance: Essential for Unified Namespace (UNS) architectures where decoupled producers and consumers have independent lifecycle management.
• Trade-off: Restricts the speed of schema evolution in exchange for maximum operational flexibility.

A modification that violates the compatibility contract, causing immediate deserialization failures. These must be managed explicitly.

• Examples of Breaking Changes:
  • Renaming a field (e.g., temp to temperature)
  • Changing a field's data type (e.g., int to float)
  • Deleting a required field
  • Adding a new required field without a default
• Mitigation Strategy: Requires a two-phase upgrade—first deploy a new compatible schema with both fields, migrate consumers, then remove the old field in a subsequent release.
• Detection: A Schema Registry enforces these rules and rejects incompatible registrations at the producer level.

A stricter variant that ensures a new schema is compatible not just with the immediately preceding version, but with all previously registered versions in the subject's history.

• Why It Matters: Prevents a scenario where gradual, individually compatible changes accumulate into a state that breaks very old consumers that haven't been updated in months.
• Industrial Context: Critical for Data Historians and long-term archival systems where data written years ago must still be queryable by current analytics tools.
• Enforcement: Configured as a global or subject-level rule in the schema registry to guarantee universal readability across the entire data lineage.

A mechanism to compose complex schemas from reusable building blocks, avoiding monolithic definitions and managing evolution at a granular level.

• How It Works: A top-level schema can reference a child schema (e.g., a motor_metrics record) stored independently in the registry.
• Evolution Benefit: The child schema can evolve independently under its own compatibility rules without forcing a version bump on every parent schema that includes it.
• Example: A production_line_status schema references a temperature_reading schema and a vibration_reading schema. Updating the vibration_reading definition does not require re-registering the entire line status schema.

A centralized service that stores and manages the schemas for data formats like Avro or Protobuf. It acts as the single source of truth, ensuring that producers and consumers can reliably deserialize streaming data. The registry enforces compatibility rules and stores a versioned history of every schema change.

• Stores Avro, Protobuf, and JSON Schema definitions
• Enforces Forward, Backward, and Full compatibility modes
• Critical for decoupled producer-consumer evolution in Apache Kafka pipelines