Schema Evolution

A core principle governing how schema changes affect existing data and applications.

• Backward Compatibility: Newer schema versions can read data written under older schemas. This is often managed through default values for new properties or ignoring unknown fields.
• Forward Compatibility: Older application versions can read data written under newer schemas, typically requiring the schema to be additive only (e.g., only adding optional properties or new vertex/edge types).
• Breaking changes, like renaming a property key or changing a data type, require a coordinated data migration strategy to avoid application errors.

Established techniques for applying and managing schema changes over a live dataset.

• Expand-and-Contract: A safe, multi-phase pattern. First, expand the schema (add new property). Second, migrate all data and update application logic to use the new property. Finally, contract by removing the old, deprecated property.
• Dual-Writing: During a transition, applications write data to both the old and new schema structures. This allows for a gradual rollout and verification.
• Versioned Schemas: Assigning explicit version numbers to schemas, allowing different parts of the graph or different queries to reference specific versions. This is common in RDF using named graphs.

Graph databases occupy different points on a spectrum from rigid to flexible schema enforcement.

• Schema-on-Write (Rigid): Requires a predefined schema (e.g., vertex schema, edge schema) with constraints before data can be inserted. Changes require explicit schema migration. Ensures high data quality.
• Schema-on-Read (Flexible): Allows data to be written with any structure; the schema is interpreted at query time. Enforcement happens via application logic or tools like SHACL for validation.
• Hybrid Approaches: Many property graph databases use a schema-optional or schema-mixed model, where some labels have strict schemas while others are flexible, offering a balance of control and agility.

Schema evolution must carefully manage defined rules to prevent data corruption.

• Uniqueness Constraints: Adding a uniqueness constraint on an existing property requires verifying no duplicates exist in the current data.
• Cardinality Constraints: Changing a relationship's cardinality (e.g., from MANY to ONE) requires ensuring the existing graph data complies with the new rule.
• Data Type Changes: Altering a property's data type (e.g., String to Integer) necessitates a transformation of all existing values, which can be a costly operation.
• Mandatory Properties: Making a previously optional property mandatory requires populating it for all existing vertices/edges of that type.

Software and processes that support safe, repeatable schema changes.

• Schema Migration Scripts: Idempotent scripts (e.g., in Cypher or Gremlin) that codify each schema change, often managed with version control.
• Schema Diff & Comparison: Tools that analyze differences between two schema versions to auto-generate migration scripts or impact reports.
• Continuous Integration for Schema: Integrating schema validation and migration testing into CI/CD pipelines to catch breaking changes early.
• Runtime Schema Validation: Using SHACL shapes or database-native constraints to validate data integrity continuously, even as the schema evolves.

The philosophical and technical differences in how two major graph paradigms handle change.

• RDF & Ontology Evolution: Focuses on logical consistency within an ontology (defined with RDFS or OWL). Changes involve adding/removing axioms (e.g., rdfs:subClassOf). Reasoning engines must handle monotonic vs. non-monotonic changes. SHACL is used for structural validation separate from ontological reasoning.
• Property Graph Evolution: Focuses on the structural schema for labels and properties. Changes are often more operational (add label, index property). Native constraints (uniqueness, cardinality) are common. Evolution is typically managed through database-specific APIs or query language commands.

Introduces a new category of entity to the graph. This is a forward-compatible operation, as existing data and queries are unaffected.

• Example: Adding a Product vertex type to an existing customer graph to support e-commerce features.
• Implementation: Typically involves defining a new label and its associated property keys and constraints. New vertices can be created immediately, and existing vertices cannot be retroactively assigned this new label without a data migration.

Defines a new type of connection between existing or new vertex types. This expands the relationship ontology of the graph.

• Example: Adding a WORKS_WITH edge between Employee vertices to model internal collaboration networks.
• Considerations: Must specify valid source and target vertex types. Adding a new relationship type is generally safe, but queries expecting the old schema will not traverse these new paths unless updated.

Extends a vertex or edge type with a new attribute. This is a common and typically non-breaking change.

• Example: Adding a middleName property to the Person vertex type.
• Schema-on-Write vs. Schema-on-Read: In strict schema-on-write systems, the property key must be defined before use. In schema-on-read or flexible systems, properties can be added dynamically, but defining them in the schema improves validation and indexing.
• Backfilling: Existing vertices/edges will have a null or absent value for the new property unless a data migration populates it.

Changes the expected data type (e.g., String to Integer) for an existing property key. This is a breaking change that requires careful data migration.

• Risk: Existing data of the old type will cause errors if not transformed. Queries and application code relying on the old type will fail.
• Process: Requires a multi-step strategy:
  1. Create a new temporary property key with the new type.
  2. Migrate and transform data from the old key to the new key.
  3. Update all application code and queries to use the new key.
  4. Remove the old property key (optional).

Enforces data integrity rules, such as uniqueness or cardinality. Adding constraints validates future writes; modifying them can affect existing data.

• Uniqueness Constraint: Ensures a property's value is unique across a vertex/edge type (e.g., email on User). Adding this fails if duplicate values already exist.
• Cardinality Constraint: Restricts the number of relationships of a type a vertex can have (e.g., a Person can have only one HAS_SSN edge).
• Existence Constraint: Mandates that a property must have a non-null value. Adding this fails for any existing vertex/edge missing that property.

The process of marking vertex types, edge types, or properties as obsolete for future use while maintaining backward compatibility for a transition period.

• Strategy:
  • Documentation: Clearly mark the element as deprecated in the schema registry.
  • Write Restrictions: Prevent new data from using the deprecated element, but allow reads.
  • Data Migration: Gradually migrate existing data that uses the deprecated element to the new standard.
  • Removal: After the migration is complete and a grace period has passed, the element can be removed from the schema, which is a breaking change.
• Purpose: Allows for clean, controlled schema refactoring without immediate system disruption.

A flexible data modeling paradigm where the structure (schema) is interpreted and applied at query time, not enforced at ingestion. This contrasts with schema evolution's focus on modifying a defined structure.

• Key Benefit: Enables rapid ingestion of heterogeneous or semi-structured data without upfront modeling.
• Trade-off: Defers data quality and consistency checks, potentially leading to query-time errors.
• Use Case: Ideal for exploratory data analysis or integrating data lakes where the structure is unknown or highly variable.

A rigid data modeling approach where data must conform to a predefined schema before it is written to the database. Schema evolution is the process of safely changing this enforced schema over time.

• Key Benefit: Guarantees data integrity, consistency, and quality at the point of ingestion.
• Foundation for Evolution: Provides the stable baseline structure that subsequent schema changes modify.
• Use Case: Essential for production systems requiring strong data governance, predictable performance, and reliable applications.

A W3C standard language for validating RDF graphs against a set of conditions called Shapes. It is a primary tool for enforcing schema integrity during and after evolution in semantic graphs.

• Validation Role: Defines expected property types, cardinality, and value ranges for nodes.
• Evolution Support: SHACL shapes themselves must be versioned and migrated alongside data.
• Core Function: Provides a declarative way to test whether graph data conforms to the evolved schema, catching violations early.

A schema rule that ensures the value of a specified property is unique across all vertices or edges of a given type. Managing these constraints is a critical aspect of schema evolution.

• Evolution Challenge: Adding or modifying a uniqueness constraint on existing data requires verifying no duplicates exist.
• Migration Impact: May necessitate data cleanup scripts before the new schema version can be applied.
• Example: Evolving a schema to enforce a unique employeeId on all Person vertices.

A schema rule that restricts the number of relationships of a specific type a vertex can have (e.g., Person OWNS exactly one PrimaryPassport). Evolution often involves relaxing or tightening these constraints.

• Types: Includes one-to-one, one-to-many, and many-to-many restrictions.
• Evolution Scenario: Changing a WORKS_FOR edge from many-to-one (one employer) to many-to-many (multiple concurrent employers) to reflect new business rules.
• Data Migration: Existing data may need to be reviewed or transformed to comply with a new cardinality rule.

The process of defining transformation rules between a source data schema and a target graph schema. It is a prerequisite activity for data migration during a schema evolution step.

• Evolution Link: When a graph schema evolves, existing schema mappings from upstream sources (e.g., CRM, ERP) must be updated to align with the new target structure.
• Automation: Often implemented using ETL/ELT tools or declarative mapping languages.
• Goal: Ensures data from external systems is correctly transformed and loaded into the new, evolved graph structure.