Schema Evolution

A schema change is backward compatible if new data written with the updated schema can be read by consumers using the older version of the schema. This is the most common requirement for safe evolution.

• Example: Adding a new optional column to a table. Old consumers can still read new data, ignoring the new column.
• Critical For: Rolling updates where producers and consumers are updated at different times.
• Mechanisms: Using optional fields, providing sensible defaults for new fields, and avoiding renaming or deleting required fields.

A schema change is forward compatible if old data written with a previous schema can be read by consumers using the newer version of the schema. This protects against "reader" rollbacks.

• Example: Removing an optional column. New consumers can still read old data that contained the now-removed column.
• Critical For: Disaster recovery scenarios where a new service might need to be rolled back to an older version that must read recently written data.
• Mechanisms: Using schema evolution rules that ignore unknown fields (like in Protobuf or Avro) and avoiding making previously optional fields required.

These are two fundamental paradigms for applying schema during data processing.

• Schema-on-Write: The schema is enforced when data is ingested or stored (e.g., in a traditional RDBMS or data warehouse). Evolution requires explicit migration (ALTER TABLE). Provides early validation but less flexibility.
• Schema-on-Read: The schema is applied when data is queried or consumed (common in data lakes). The raw data is stored flexibly, and the schema is interpreted by the reading application (e.g., Spark, Trino). Enables easier exploration but shifts validation burden downstream.
• Modern architectures like the Data Lakehouse (e.g., Delta Lake, Apache Iceberg) blend these by storing data flexibly but providing transactional schema enforcement and evolution capabilities on top.

Practical patterns for implementing schema changes with minimal disruption.

• Additive Changes: The safest method. Only add new optional fields or columns. Never remove or rename existing ones.
• Expand-and-Contract (Parallel Change): A multi-phase pattern: 1) Expand schema to support both old and new structures, 2) Update all producers and consumers to use the new structure, 3) Contract by removing the old structure once it's unused.
• Default Values: Essential for adding new required fields. The schema defines a default value that is applied when reading old data that lacks the field.
• Deprecation: Mark fields as deprecated in the schema metadata, communicate timelines to consumers, and schedule removal for a future version after a grace period.

A formal agreement between a data producer and its consumers that codifies the schema, semantics, quality, and service-level expectations (e.g., freshness, latency) for a data product.

• Enforces Evolution Policies: The contract explicitly states the allowed compatibility modes (e.g., "backward compatible only") and change notification procedures.
• Beyond Schema: Includes commitments on data quality metrics (null rates, uniqueness), SLAs, and deprecation policies.
• Automated Enforcement: Contracts can be validated as part of CI/CD pipelines and pipeline monitoring, preventing breaking changes from being deployed.

Backward compatibility ensures new schema versions can read data written with old schemas. Forward compatibility ensures old schema versions can read data written with new schemas. These principles are critical for zero-downtime deployments and rolling updates.

• Example: Adding an optional field with a default value is backward compatible. Removing a required field is not.
• Common Patterns: Using optional fields, providing sensible defaults, and employing schema evolution-aware serialization formats like Avro, Protobuf, or JSON Schema.

These are two fundamental approaches to schema enforcement. Schema-on-Write validates and enforces a schema when data is ingested (e.g., traditional databases, data warehouses). Schema-on-Read applies a schema when data is queried, offering flexibility for data lakes.

• Schema-on-Write: Ensures high data quality at ingestion but can be rigid. Changes often require expensive migrations.
• Schema-on-Read: Allows storing diverse, raw data but shifts quality and transformation costs to consumers. Modern data lakehouse architectures blend both approaches.

Schema changes are categorized by their impact on compatibility and the required migration strategy.

• Additive Changes: Adding a new optional column or field. This is generally safe and both backward and forward compatible.
• Subtractive Changes: Removing a field. This breaks backward compatibility and requires careful orchestration, often using deprecation flags first.
• Transformative Changes: Modifying a field's data type (e.g., INT to BIGINT) or constraints. This typically requires a data migration script and a coordinated update of producers and consumers.

A Data Contract is a formal, versioned agreement between data producers and consumers. It codifies the schema, semantics, quality SLAs (freshness, completeness), and deprecation policies.

• Purpose: Moves schema management from an ad-hoc process to a governed, product-oriented practice. It explicitly defines breaking vs. non-breaking changes.
• Components: Includes the technical schema, examples, ownership, and service-level objectives (SLOs).
• Benefit: Enables safe schema evolution by providing consumers with clear expectations and advance notice of changes.

Managing schema evolution at scale requires automation integrated into CI/CD and data pipelines.

• Migration Frameworks: Tools like Liquibase or Flyway manage incremental, versioned schema changes for SQL databases, applying them as part of deployments.
• Pipeline Orchestration: Data pipeline tools (e.g., Apache Airflow, dbt) can execute data transformation jobs as part of a schema change, backfilling new columns or converting data types.
• Observability Integration: Changes should trigger updates to data lineage maps and be monitored via data quality checks to catch downstream breakage.

A centralized service that stores and manages the schemas (e.g., Avro, Protobuf, JSON Schema) for data in motion, typically within event streaming platforms like Apache Kafka. It enforces compatibility rules (backward, forward, full) to ensure that schema changes do not break existing producers or consumers, providing a single source of truth for serialization and deserialization.

• Key Function: Validates schema changes against a defined compatibility policy.
• Use Case: Critical for decoupling services in a microservices architecture where data formats evolve independently.

A formal, versioned agreement between a data producer and one or more data consumers. It explicitly defines the expected interface for a data product, including:

• Schema: The exact structure, data types, and constraints.
• Semantics: The meaning of fields and allowable values.
• Service-Level Objectives (SLOs): Guarantees for freshness, latency, and availability.
• Evolution Rules: Policies for how the contract can change (e.g., notification periods, deprecation timelines).

Data contracts make schema evolution a predictable, managed process, reducing breaking changes.

Two fundamental compatibility modes that dictate how systems handle schema changes.

• Backward Compatibility: A new schema can read data written with an old schema. This is achieved by adding optional fields or providing default values for new required fields. Consumers can upgrade first.
• Forward Compatibility: An old schema can read data written with a new schema. This is achieved by ignoring unknown fields. Producers can upgrade first.

Full Compatibility requires both. Choosing the right mode is a strategic decision for rolling updates in distributed systems.

A design pattern that identifies and captures incremental changes (inserts, updates, deletes) made to data in a source database. CDC is a primary enabler for real-time data replication and is deeply intertwined with schema evolution.

• Impact on Evolution: CDC tools must handle source schema changes (e.g., new columns) and propagate them to downstream consumers without data loss or pipeline failure.
• Implementation: Often uses database transaction logs. Tools include Debezium, AWS DMS, and Striim.

The granular tracking of data flow and transformation at the level of individual columns, from original source to final consumption. It is critical for impact analysis during schema evolution.

• Use in Evolution: Before renaming or deleting a column, engineers can trace all downstream dependencies (reports, models, applications) to assess the blast radius and coordinate changes.
• Contrast with Table-Level: Provides much finer-grained visibility, essential for complex, wide datasets.

A centralized repository that documents the technical metadata for data elements within a specific database, file, or system. It is a foundational tool for managing schema understanding.

• Contents: Includes precise definitions for tables, columns, data types, constraints, primary/foreign keys, and allowed values.
• Role in Evolution: Serves as the authoritative source for the current state of a schema. Changes proposed during evolution should be reflected here first to maintain accurate documentation.
• Contrast with Business Glossary: Focuses on technical attributes rather than business meaning.