Data Contract

The schema is the formal, machine-readable specification of the data's structure. It defines the data types, field names, nested structures, and constraints for every record. For multimodal data, this often involves complex, nested schemas to represent different data types (e.g., a video stream contract might define fields for frame rate, resolution, codec, and a nested array for audio track metadata). Common formats include JSON Schema, Apache Avro, and Protocol Buffers (Protobuf).

This component defines the meaning and context of the data beyond its raw structure. It answers the "what" and "why." Key elements include:

• Business Glossary Terms: Links fields to standardized business definitions (e.g., customer_id maps to the enterprise definition of a customer).
• Data Classification: Tags for sensitivity (PII, PHI, confidential).
• Modality-Specific Context: For a sensor data contract, this could specify the unit of measurement (e.g., acceleration_x: m/s^2), sensor calibration parameters, or the coordinate system for 3D point clouds.
• Temporal Context: Defines if data is a snapshot, event stream, or time-series, and the meaning of timestamps.

The contract specifies operational guarantees for the data product. These are quantifiable Service Level Objectives (SLOs) that form the basis of a Service Level Agreement (SLA). Critical metrics include:

• Freshness/Latency: Maximum allowed delay between data creation and availability (e.g., "95% of records delivered within 2 seconds").
• Availability: Uptime percentage for the data stream or endpoint.
• Throughput: Expected volume of records per second/minute.
• Schema Stability Guarantee: Rules for schema evolution, such as a commitment to only make backward-compatible changes (e.g., adding optional fields) without breaking consumers.

Explicit, testable assertions about the content and characteristics of the data. These rules move beyond "structure" to define "fitness for use." They are often expressed as SQL-like checks or using frameworks like Great Expectations. Examples include:

• Completeness: "Field sensor_reading shall not be null for >99.9% of records."
• Validity: "user_age must be an integer between 0 and 120."
• Accuracy/Plausibility: "gps_latitude must be within bounds [-90, 90]."
• Uniqueness: "transaction_id must be unique per day."
• Statistical Rules: "The mean value of temperature_c for a sensor batch must be within 3 standard deviations of the rolling 24-hour average."

Defines the technical mechanism for data exchange. This removes ambiguity about how to get the data. Specifications include:

• Protocol: The communication standard (e.g., HTTP/REST, gRPC, Apache Kafka, MQTT for IoT).
• Endpoint/ Topic: The specific URL, queue, or Kafka topic name.
• Authentication & Authorization: Required credentials, API keys, or OAuth scopes.
• Serialization Format: The on-the-wire format (e.g., Avro binary, JSON, Protobuf).
• Compression: If applicable (e.g., gzip, snappy).

This component establishes accountability and traceability.

• Data Product Owner: The specific team or system (the producer) responsible for upholding the contract.
• Consumer Registration: A mechanism for downstream systems (the consumers) to declare their dependency, enabling change management and impact analysis.
• Data Lineage: References to upstream sources. For a derived data product, the contract should specify the source datasets and transformation logic, often linking to pipeline code (e.g., a Git commit hash) or an Apache Airflow DAG ID. This is critical for debugging and compliance.

The practice of managing changes to a data schema over time while maintaining compatibility with existing data and applications. It is a core requirement for a durable data contract, governed by rules like:

• Backward Compatibility: New schema can read data written with the old schema (e.g., adding an optional field).
• Forward Compatibility: Old schema can read data written with the new schema (e.g., removing an optional field).
• Full Compatibility: Both backward and forward compatibility are maintained.

Evolution strategies (ADD, DELETE, RENAME) must be explicitly defined in the contract to prevent pipeline breaks during updates.

A measurable target for a specific aspect of a service's performance. In a data contract, SLOs define the non-functional guarantees provided by the data product. Critical SLOs for data contracts include:

• Freshness/Latency: Data is available for consumption within a defined time window (e.g., < 5 minutes from event time).
• Availability: The data stream or endpoint is accessible a defined percentage of the time (e.g., 99.9%).
• Correctness/Accuracy: Data conforms to schema and quality rules (e.g., null rate for a field < 0.1%).
• Throughput: The system can handle a defined volume of data (e.g., 10,000 events/sec).

Breaching an SLO constitutes a contract violation, triggering alerts and remediation.

The tracking of data's origin, movement, transformation, and dependencies across its lifecycle. It provides the audit trail and impact analysis for a data contract by answering:

• Provenance: Where did this dataset originate? Which upstream contracts does it depend on?
• Transformation: How was the data derived or changed?
• Consumption: Which downstream models, applications, or reports use this data?

Impact: When a data contract changes (schema evolution), lineage tools identify all affected downstream consumers, enabling proactive communication and testing—a key governance function.

The systematic, automated monitoring of data to detect anomalies, drifts, and lineage breaks. It is the continuous validation mechanism for a data contract's guarantees. Key monitoring dimensions include:

• Volume: Unexpected spikes or drops in data flow.
• Schema: Drift from the contracted schema.
• Freshness: Data arriving later than the SLO.
• Distribution: Statistical data drift in key field values (e.g., average transaction amount shifts).
• Integrity: Violations of business rules (e.g., foreign key constraints, value ranges).

Tools in this space (e.g., Monte Carlo, Great Expectations) run automated checks and alert on contract violations before they degrade downstream systems.