Data Contract

The core of a data contract is a precise, versioned definition of the data's structure. This includes:

• Column names and data types (e.g., user_id: UUID, transaction_amount: DECIMAL(10,2))
• Constraints like nullability, uniqueness, and primary/foreign keys
• Nested structures for semi-structured data (JSON, Avro, Protobuf)

This schema acts as the source of truth for serialization/deserialization and is validated at ingestion.

Contracts embed executable data quality rules that define acceptable data states. These go beyond schema to include:

• Statistical boundaries: Value ranges, standard deviation thresholds
• Business logic: order_total must equal sum of line_item_totals
• Freshness SLAs: Maximum allowed latency from source event to data availability
• Volume expectations: Expected row count ranges per update

Violations trigger alerts or block pipeline progression, enforcing the contract.

The contract specifies how data is accessed and its evolution rules:

• Access patterns: API endpoints, query interfaces, or direct file paths (e.g., s3://bucket/table/{date}/)
• Serialization format: Avro, Parquet, JSON Schema
• Backward/forward compatibility policies: Rules for safe schema evolution (e.g., only adding optional fields)
• Deprecation notices: Timelines for removing fields or changing types

This prevents breaking changes from silently affecting downstream consumers.

Contracts include operational and provenance metadata:

• Ownership: Data product owner, team, and contact
• Lineage: Upstream sources and downstream dependencies
• Sensitivity classification: PII, financial data, public data
• Refresh frequency and timing: Batch schedules or streaming latency
• Sample data: Representative examples of valid records

This context is crucial for discovery, governance, and impact analysis during incidents.

A contract is not a document but enforced code. Implementation involves:

• Validation engines: Tools like Great Expectations, dbt tests, or custom validators that run checks at pipeline stages
• Pipeline gates: Fail CI/CD builds or halt production pipelines on contract violation
• Consumer verification: SDKs or libraries that allow consumer applications to validate data against the contract locally
• Versioned artifacts: Contracts are stored in version control (Git) and published to a registry, enabling dependency management.

A concrete example for an orders table contract:

yamlCopy
schema:
  fields:
    - name: order_id
      type: string
      constraints: [unique, not_null]
    - name: customer_id
      type: string
      constraints: [not_null]
    - name: amount
      type: float
      constraints: [not_null]
quality_rules:
  - rule: amount > 0
    severity: error
  - rule: freshness < 1 hour
    severity: warning
metadata:
  owner: [email protected]
  refresh: hourly
  source: payment_service_db

This YAML would be compiled into validation code and enforced at each pipeline run.

The core of a data contract is a machine-readable schema that defines the expected structure. Common tools and formats include:

• Avro, Protobuf, or JSON Schema: Provide strong typing and serialization for data at rest or in motion.
• Pydantic (Python) or TypedDict: Enable runtime data validation and parsing using Python type hints.
• SQL DDL: Traditional Data Definition Language statements that enforce structure directly in databases. These schemas act as the single source of truth for the data's shape, enabling automatic validation at pipeline ingress points.

Specialized platforms treat the contract as version-controlled, executable code integrated into CI/CD pipelines.

• Great Expectations: Creates Expectation Suites that codify rules for data quality, schema, and freshness, which can be executed as pipeline-gated tests.
• dbt: Uses dbt tests (defined in YAML or SQL) to enforce contracts on transformed data models, validating uniqueness, referential integrity, and custom business logic.
• Soda Core: Employs a YAML-based scan definition to specify checks for volume, freshness, schema drift, and custom metrics, functioning as a contract validation engine.

For real-time data products, contracts are enforced at the message broker level.

• Apache Kafka with Schema Registry: A central registry (e.g., Confluent Schema Registry) manages Avro, JSON Schema, or Protobuf schemas. Producers and consumers must use compatible schemas, preventing breaking changes from propagating.
• Contract Testing in Event-Driven Architectures: Tools like Pact can be adapted to verify that event producers and consumers adhere to agreed-upon message formats, ensuring backward and forward compatibility.

Frameworks that treat data assets as queryable products with published interfaces.

• gRPC + Protobuf: Defines strict service and message contracts for high-performance data serving, ensuring type safety between services.
• GraphQL: Allows data consumers to define the exact shape and fields of the data they need via a strongly-typed schema, making the contract explicit and consumer-driven.
• REST APIs with OpenAPI/Swagger: While traditionally for applications, these can formally specify endpoints for accessing curated datasets, documenting structure and semantics.

Contracts are published to and discovered via a central data catalog, making them part of the organization's data fabric.

• DataHub or Amundsen: Allow attaching schema, ownership, freshness SLAs, and quality metrics to datasets as metadata. This makes the contract discoverable and provides lineage back to the source.
• Programmatic Access: Catalogs often provide APIs or SDKs (Python, Java) that allow pipeline code to automatically retrieve the latest contract schema for a dataset before reading or writing, ensuring runtime compliance.

Contract validation is automated within engineering workflows.

• Pipeline-Gated Execution: In tools like Apache Airflow, Prefect, or Dagster, a task runs the contract validation suite (e.g., a Great Expectations checkpoint). Failure prevents downstream tasks from executing, blocking bad data.
• Merge Request Checks: Contract schema files (.avsc, .proto) and test definitions are stored in Git. CI pipelines run validation tests on sample data or in a test environment, preventing breaking changes from being merged.
• Dynamic Quality Gates: Orchestrators can be configured to route data based on contract validation results—e.g., sending failing records to a quarantine bucket for analysis.

A declarative or programmatic statement that defines a condition data must satisfy to be considered valid. This is the atomic unit of a data contract.

• Examples: A column must be non-null, values must fall within a predefined range (e.g., age BETWEEN 0 AND 120), or a string must match a specific regex pattern.
• Role in Contracts: Data contracts are composed of multiple, coordinated data quality rules that collectively define the schema, freshness, and semantic guarantees of a data product.

A programmatic check embedded within a data pipeline that actively verifies a specific condition about the data and raises an error or warning if the condition is false. It is the execution mechanism for a data quality rule.

• Key Function: Acts as a runtime guard. For example, an assertion might check that the row count of a daily batch is within 10% of the historical average before allowing the pipeline to proceed.
• Contract Enforcement: Data contracts rely on assertions to be operational. A contract stating "data must be delivered by 9 AM UTC" would have a corresponding assertion that fails if the data asset's timestamp is later than that.

A predefined quality threshold or set of passing tests that must be met for data to progress from one stage of a pipeline to another or to be deemed fit for consumption. It is a control point that enforces a data contract.

• Pipeline-Gated Tests: These are assertions whose failure prevents downstream processing, physically blocking bad data. This is a direct implementation of a contract's "guarantee."
• Usage: A gate might require all schema validation rules and 95% of business rule validations to pass before data is published to a production analytics table, ensuring consumers only receive data that meets the contracted standards.

An approach to data quality where tests are specified as configuration stating what the data condition should be, rather than how to check it programmatically. This paradigm is essential for scalable, maintainable data contracts.

• Manifestation: Rules are defined in YAML, SQL, or a domain-specific language (e.g., column: customer_id, tests: [unique, not_null]).
• Advantage for Contracts: It separates the definition of the agreement (the contract's terms) from the implementation of the validation logic, making contracts easier to version, share, and understand by both producers and consumers.

The process of verifying that data conforms to expected formats, types, and structural rules. This is the most fundamental layer of any data contract.

• Schema Validation: Ensures the presence, name, and data type of columns (e.g., using JSON Schema, Protobuf, or Avro). This is often the first check in a contract.
• Data Validation: Ensures the actual values within those columns adhere to semantic rules (e.g., an email column contains valid email addresses, a status column contains only enumerated values).
• Together, they form the technical specification that prevents consumers from experiencing schema-on-read errors or corrupted analyses.