Data Mesh

Data ownership and architecture are decentralized to business domains (e.g., Finance, Logistics, Customer Service). Domain teams become responsible for their data as a product, managing its quality, documentation, and lifecycle. This replaces the centralized data team bottleneck.

• Key Shift: From centralized data lakes/warehouses owned by IT to distributed data products owned by business units.
• Example: The e-commerce 'Checkout' domain team owns and serves the order_events data product.

Domain data is treated as a consumable product with explicit service-level objectives (SLOs), documentation, and a defined interface. This ensures discoverability, trustworthiness, and usability for internal consumers.

• Core Components: Must include documentation, a guaranteed SLA/SLO, and a standardized interface (e.g., an API or a published schema).
• Product Thinking: Applies software product management principles—like user empathy and iterative improvement—to data assets.

A federated, interoperable platform provides domain teams with the tools and abstractions needed to build, deploy, and manage their data products autonomously. It standardizes infrastructure without centralizing control.

• Platform Capabilities: Provides standardized services for data storage, pipeline orchestration, metadata cataloging, and access control.
• Goal: Reduces the cognitive load and time-to-value for domain teams, enabling them to act as independent data product developers.

A global governance model defines interoperability standards, security policies, and compliance rules, which are then enforced by automated, code-based policies within the self-serve platform.

• Federated Model: A cross-domain governance group sets global standards (e.g., for data classification, PII handling), while domains retain autonomy in implementation.
• Computational Enforcement: Policies are codified and automated (e.g., via schema registries, access control lists) rather than enforced through manual reviews.

The core principle of Data Mesh, treating data assets as first-class products with explicit service-level agreements (SLAs), clear ownership by domain teams, and a focus on discoverability, security, and usability for internal customers.

• Key Components: A documented schema, guaranteed quality, a defined lineage, and a discoverable interface (e.g., a data catalog entry).
• Contrasts with treating data as a by-product of applications, where quality and accessibility are secondary concerns.

The organizational principle of aligning data ownership and architecture with business domains (e.g., customer, inventory, finance) rather than centralizing it in a singular data team.

• Domain Data Teams: Cross-functional teams own their domain's data products end-to-end, from generation to serving.
• Reduces Bottlenecks: Eliminates the dependency on a central data engineering team for all new data needs, accelerating time-to-insight.
• Requires a shift in organizational structure and the establishment of federated computational governance.

The enabling infrastructure layer that provides domain teams with the standardized tools and services needed to build, deploy, and manage their data products autonomously.

• Abstracts Complexity: Provides managed services for storage, computation, pipeline orchestration, and cataloging so domain engineers don't need deep data infrastructure expertise.
• Key Capabilities: Automated data product deployment, unified access control, monitoring, and global discovery through a data catalog.
• Analogy: Similar to how AWS enables application teams by providing managed databases and compute, a self-serve data platform does this for data products.

A model of decentralized decision-making and policy enforcement where global standards (for interoperability, security, compliance) are defined centrally but executed locally by the self-serve platform and domain teams.

• Computational Policies: Standards (e.g., PII tagging, encryption) are encoded as code and automatically enforced by the platform, not just documented.
• Balances Autonomy & Control: Enables domain autonomy while ensuring cross-domain data products can be reliably composed and used.
• Examples: Platform-enforced schema validation, automatic lineage tracking, and standardized access control models.

The atomic, addressable unit of data in a Data Mesh. It is a node in the mesh that encapsulates data, its code, metadata, and policies, served via standardized interfaces.

• Characteristics: Discoverable, Addressable, Trustworthy, Self-Describing, Interoperable, and Secure.
• Interfaces: Typically provides both batch (e.g., files in cloud storage) and event-stream (e.g., Kafka topic) access patterns.
• Ownership: A single domain team is the source of truth and responsible for its lifecycle, quality, and SLAs.

A modern data architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance management of a data warehouse. It is a common target storage layer for data products in a Data Mesh implementation.

• Supports Data Mesh: Provides a unified storage foundation where domain teams can manage their data products while enabling efficient cross-domain querying and analytics.
• Key Technologies: Apache Iceberg, Apache Hudi, and Delta Lake provide the table format that enables these capabilities on object storage like Amazon S3.
• Contrast: Unlike a monolithic data warehouse, a lakehouse in a mesh is a platform capability, not a centrally managed monolithic dataset.