Data Lakehouse

Provides Atomicity, Consistency, Isolation, and Durability guarantees on data lake storage. This allows multiple concurrent readers and writers to operate safely without corrupting data, enabling concurrent ML training jobs and BI reporting on the same live data. Technologies like Apache Iceberg and Delta Lake implement optimistic concurrency control to serialize transactions.

Validates all incoming data against a defined schema, rejecting corrupt or malformed records at write time. Simultaneously supports safe schema evolution—allowing columns to be added, dropped, or renamed without breaking downstream consumers. This prevents the 'data swamp' problem common in traditional data lakes while maintaining the flexibility to adapt to changing data sources.

Handles both batch processing and real-time streaming on a single storage layer. The same table can serve as a sink for streaming ingestion and a source for batch model training. This eliminates the need for complex lambda architectures and ensures that ML models are trained on the same consistent data that powers real-time dashboards.

Maintains immutable snapshots of data at every transaction commit. Enables point-in-time queries to reproduce exactly the dataset used for a specific model training run. Critical for regulatory audits, debugging model performance regressions, and rolling back to a known-good state after a data pipeline failure.

Stores data in open, vendor-neutral formats like Apache Parquet and ORC. This prevents vendor lock-in and allows multiple compute engines—Spark, Trino, Presto, Dremio—to operate on the same data without proprietary translation layers. ML frameworks can read directly from the lakehouse without ETL extraction.

Implements row-level and column-level security directly on the storage layer. Data engineers can define policies that mask or filter sensitive columns for specific user roles. This allows a single table to serve both a compliance officer who sees all data and a data scientist who sees only anonymized features.

A decentralized sociotechnical architecture that organizes data by business domain, treating data as a product owned by the domain team that creates it. Unlike a monolithic lakehouse, a data mesh distributes ownership and governance.

• Shifts responsibility to domain experts
• Enforces global interoperability standards
• Reduces central bottleneck dependencies

An enterprise storage strategy using multiple database technologies—relational, graph, document, and vector stores—within a single application or architecture. A lakehouse often serves as the unifying layer over polyglot persistence.

• Uses the right tool for each data shape
• Avoids forcing all data into one model
• Requires robust metadata management

A set of software design patterns that identify and track incremental changes to source data, enabling efficient, low-latency replication into the lakehouse without full batch reloads. CDC keeps analytical views synchronized with operational systems.

• Captures inserts, updates, and deletes
• Minimizes replication latency
• Reduces load on source databases

The ability to automatically adapt a data system's structure to handle changes in incoming data formats over time without breaking downstream consumers. A critical feature of lakehouse table formats like Apache Iceberg and Delta Lake.

• Handles added, renamed, or dropped columns
• Maintains backward compatibility
• Prevents pipeline failures from schema drift

The practice of tracking and managing unique states of a dataset over time, enabling reproducible model training and rollback to previous data snapshots. Lakehouse formats provide time-travel capabilities for point-in-time queries.

• Enables audit trails for compliance
• Supports reproducible ML experiments
• Allows safe rollback of corrupted data

A centralized platform for storing, versioning, and serving curated feature data consistently across model training and low-latency inference. The lakehouse often serves as the offline storage backbone for the feature store.

• Bridges training and serving environments
• Eliminates training-serving skew
• Enables feature reuse across teams