Data Contract

The core of a data contract is a machine-readable schema that defines the exact structure, data types, and constraints of the data product. This is not just a list of columns, but a formal specification that can be used for automated validation.

• Enforced Structure: Defines tables, columns, nested fields, and primary/foreign keys.
• Type Safety: Specifies precise data types (e.g., timestamp with timezone, decimal(10,2), array<string>).
• Constraints: Includes rules like non-nullability, uniqueness, value ranges, and regex patterns for strings.
• Common Formats: Often expressed in Avro, Protobuf, or JSON Schema, which support schema evolution rules (e.g., backward/forward compatibility).

Beyond structure, a contract provides the business meaning of the data. This bridges the gap between technical implementation and business understanding, ensuring data is used correctly.

• Column Definitions: Plain-language descriptions of what each field represents (e.g., customer_lifetime_value is "The total net profit attributed to the customer over their entire relationship, calculated quarterly").
• Business Glossary Links: References standardized terms from an organizational business glossary.
• Metric Definitions: Explicit formulas for calculated fields and key performance indicators.
• Usage Examples: Sample values and scenarios illustrating correct interpretation.

The contract specifies non-functional requirements that define the reliability and performance characteristics of the data product, similar to SLAs for software services.

• Freshness & Latency: Maximum allowed data delay (e.g., "data is updated within 5 minutes of source event").
• Availability & Uptime: Expected availability percentage for data delivery.
• Volume & Throughput: Expected row counts or data size per update.
• Error Budgets: Allowable amount of downtime or quality violations before a breach is declared.
• Support & Incident Response: Defined channels and response times for issue reporting.

Explicit, testable assertions about the content and integrity of the data. These rules are enforced at the point of production and can be monitored by consumers.

• Completeness: Rules specifying allowed thresholds for null values.
• Accuracy: Rules validating data against a source of truth or logical constraints (e.g., end_date must be after start_date).
• Uniqueness: Assertions that certain keys or combinations are unique.
• Consistency: Rules ensuring values are consistent across related datasets or over time.
• Statistical Rules: Expectations for value distributions (e.g., column_a values expected to follow a normal distribution with mean ~100).

A critical component that defines how the contract can change over time without breaking downstream consumers. It manages the lifecycle of the data product.

• Version Identifier: A clear, immutable version number or hash for the contract.
• Compatibility Modes: Declared rules (e.g., backward compatible, breaking change).
• Change Log: A record of all modifications, additions, and deprecations.
• Deprecation & Sunset Policy: Defined timelines for notifying consumers of upcoming breaking changes and retiring old versions.
• Governance Workflow: Process for proposing, approving, and publishing new contract versions.

Defines the human and system points of contact for the data product, enabling accountability and operational support.

• Data Product Owner: The individual or team responsible for the data's production and the contract's integrity.
• Producer System: The source application, pipeline, or service that generates the data.
• Consumer Interface: The technical endpoint for accessing the data (e.g., Kafka topic name, database table, API URL, cloud storage path).
• Sample & Documentation Links: Direct links to example data payloads and extended technical documentation.
• Metadata: Tags for data classification (e.g., PII, financial), domain, and cost center.

A data product is a reusable, high-quality data asset—such as a dataset, model, or API—that is packaged, documented, and managed as a product to serve specific consumer needs. It is the primary unit of value that a data contract governs.

• Key Characteristics: Self-describing, addressable, trustworthy, and secure.
• Contractual Interface: A data product's schema, semantics, and service levels are codified in its data contract.
• Ownership: In frameworks like Data Mesh, domain teams are responsible for the full lifecycle of their data products.

A schema registry is a centralized service that manages and stores data schemas (e.g., Avro, Protobuf, JSON Schema) for data in motion. It is a critical infrastructure component for enforcing data contracts in streaming architectures.

• Compatibility Enforcement: Validates that new schema versions are backward/forward compatible, preventing breaking changes for consumers.
• Central Reference: Provides a single source of truth for serialization and deserialization across services.
• Contract Validation: Acts as the system of record for the structural component of a data contract.

A data dictionary is a centralized repository that defines the technical attributes of data elements within a specific database or system. It focuses on the structural metadata that forms the foundation of a data contract's schema specification.

• Technical Metadata: Documents column names, data types, constraints, default values, and physical storage details.
• System-Specific Scope: Typically tied to a single database or application, unlike a broader business glossary.
• Contract Input: The precise definitions in a data dictionary are often ingested to auto-populate technical sections of a data contract.

A business glossary is a collection of definitions for key business terms, concepts, and metrics. It establishes a common organizational vocabulary, providing the semantic context for data contracts.

• Business Context: Defines what a "customer," "revenue," or "active user" means in a specific business context.
• Semantic Layer: Links technical data elements to agreed-upon business meanings, reducing misinterpretation.
• Contract Foundation: The definitions in a business glossary inform the semantic rules and quality expectations within a data contract.

Data lineage is the tracking of data's origin, movement, transformations, and dependencies across systems over its lifecycle. It provides the operational visibility needed to enforce and debug data contracts.

• Impact Analysis: Shows which downstream data products and reports will be affected if a contracted dataset changes.
• Provenance & Debugging: Traces errors in consumer data back to their source in the producer's pipeline.
• Contract Compliance: Lineage tools can validate that data is being transformed and delivered according to contracted specifications.

Data Mesh is a decentralized, domain-oriented sociotechnical framework for data architecture. It treats data as a product, making data contracts a first-class citizen for defining inter-domain interfaces.

• Domain Ownership: Business domains own, build, and serve their data as products.
• Federated Governance: Global standards (like contract schemas) are defined centrally but applied by domain teams.
• Self-Serve Platform: Provides the infrastructure (e.g., schema registries, lineage) that enables domains to create and fulfill data contracts autonomously.