Apache Iceberg

Apache Iceberg provides ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantees on object storage. This ensures data integrity by making concurrent writes safe and preventing readers from seeing partial, uncommitted data.

• Atomic commits: Changes to data files and metadata are committed as a single, instantaneous operation.
• Serializable isolation: Concurrent writers are prevented from creating conflicting table states.
• Consistent reads: Readers always see a complete, consistent snapshot of the table, even during writes.

Iceberg decouples the physical layout of data from its logical representation, enabling powerful evolution capabilities without breaking queries.

• Hidden partitioning: Queries filter on table data (e.g., WHERE event_date = '2024-01-01'), not directory paths. The table's partitioning scheme can be changed without requiring SQL queries to be rewritten.
• Safe schema evolution: Columns can be added, dropped, renamed, or have their types updated (e.g., INT to BIGINT) in a backward-compatible way. Existing data files remain valid, and readers using older schemas continue to work.

Iceberg maintains a full version history of table snapshots, enabling deterministic data auditing and recovery.

• Time travel: Query the table's state as it existed at any specific point in time or snapshot ID (e.g., SELECT * FROM table VERSION AS OF 1234 or ... TIMESTAMP AS OF '2024-01-01 10:00:00').
• Rollback: Instantly revert the entire table to a previous, known-good state. This is critical for correcting erroneous batch jobs or recovering from data corruption.

The format includes several architectural features designed for high-performance analytics on petabyte-scale datasets.

• Advanced metadata: Iceberg maintains rich metadata files (manifest lists and manifests) that catalog every data file, its partition values, and column-level statistics (min/max, null counts).
• Metadata pruning: Query engines use this metadata to skip entire files and partitions that cannot contain relevant data, drastically reducing I/O.
• Data file pruning: Within files, column-level stats enable further skipping of row groups (in Parquet/ORC).
• Partition evolution: Partition schemes can be updated to optimize for new query patterns without a costly full data rewrite.

Iceberg is designed as an open standard, independent of specific execution engines or underlying file formats.

• Multiple engine support: Tables can be created and queried by Apache Spark, Trino, Flink, Apache Hive, and many other compute engines.
• File format flexibility: Underlying data files are typically stored in efficient columnar formats like Apache Parquet, but Iceberg itself is format-agnostic.
• Object store native: It is optimized for cloud object stores (S3, ADLS, GCS) but works on HDFS and other systems.

Iceberg is a foundational component of the data lakehouse architecture, merging the best aspects of data lakes and warehouses.

• Combines strengths: It provides the low-cost, flexible storage of a data lake with the ACID compliance, schema enforcement, and performance of a data warehouse.
• Unified tier: Serves as a single, reliable source of truth for both batch and streaming data, supporting BI, SQL analytics, and machine learning workloads.
• Governance-ready: Its immutable snapshot log and rich metadata provide a strong foundation for data lineage, auditability, and governance.

Iceberg supports in-place, non-breaking schema evolution, allowing table schemas to be updated without rewriting data or breaking existing queries. Key operations include:

• Adding columns: New columns can be added and populated without affecting existing reads.
• Renaming columns: Columns can be renamed while preserving existing data; Iceberg manages the mapping.
• Evolving types: Certain type changes (e.g., int to long) are supported safely.
• Nested field evolution: Adding, removing, or renaming fields within complex structs, maps, and arrays. This eliminates costly, error-prone data migration pipelines and enables agile data product development.

Iceberg implements hidden partitioning, where the physical layout is decoupled from the logical table schema. This solves major pain points of Hive-style partitioning:

• No directory-based filters: Users query by column (e.g., WHERE event_date = '2024-01-01'), not by path. Iceberg automatically applies partition transforms.
• Partition evolution: The partition scheme of a table can be changed (e.g., from DAY(event_ts) to MONTH(event_ts)) without requiring existing data to be rewritten. New data uses the new scheme while old data remains queryable.
• Multiple partition transforms: Supports identity, bucket, truncate, year, month, day, and hour transforms on columns.

Iceberg's snapshot model enables efficient incremental processing. By tracking snapshots, systems can identify precisely what data has changed between two points in time.

• INCREMENTAL queries: Use SELECT ... FROM table CHANGES ... syntax to stream only new or modified rows.
• Downstream pipeline optimization: Downstream ETL, materialized views, or feature stores only process new data, reducing compute costs and latency.
• Merge-on-read for CDC: Efficiently apply updates from operational databases using MERGE INTO statements, which perform an upsert operation by combining new data with the existing table.

Iceberg maintains rich metadata—including manifest files with column-level statistics (min/max values, null counts)—enabling highly efficient data skipping during query planning.

• Metadata filtering: Query engines (Spark, Trino, Flink) read the metadata first to prune files that cannot contain relevant data, drastically reducing I/O.
• Automatic compaction: Small files created by streaming writes can be automatically compacted into larger files (using rewrite_data_files) to maintain optimal read performance.
• Sorting and Z-ordering: Data can be physically ordered within files using Z-order on multiple columns (e.g., user_id, event_date), co-locating related data and maximizing the effectiveness of data skipping.

Iceberg serves as a unified table layer for heterogeneous data workloads, making it a cornerstone for data lakehouse architectures.

• Unified Batch & Streaming: Serves as both the source and sink for batch (Spark) and streaming (Flink, Kafka Connect) jobs with full ACID guarantees.
• Multi-engine consistency: Tables can be written by Spark and immediately queried by Trino, Presto, or Snowflake without inconsistency, thanks to a standardized, open metadata format.
• Foundation for ML/Feature Stores: Provides reliable, versioned, and efficiently queryable storage for feature data, acting as a robust backend for feature stores. Its time-travel capability is essential for point-in-time correctness in model training.

Delta Lake is an open-source storage layer, originally developed by Databricks, that provides similar foundational capabilities to Apache Iceberg. Both are table formats designed to bring reliability to data lakes.

• Primary Comparison: Like Iceberg, Delta Lake offers ACID transactions, schema evolution, and time travel.
• Architectural Difference: While functionally similar, they differ in implementation details, metadata structure, and ecosystem integration. Iceberg is noted for its open, specification-based approach and vendor-neutral community.

A metadata catalog is a centralized registry that stores and manages metadata—such as schema, location, lineage, and access policies—for data assets. In an Iceberg architecture, the catalog tracks the current metadata pointer for each table.

• Iceberg's Requirement: Iceberg requires a catalog (e.g., AWS Glue, Hive Metastore, Nessie, JDBC) to know which metadata file represents the current, valid state of a table.
• Separation of Concerns: The catalog stores the pointer, while Iceberg's metadata files (in object storage) contain the detailed table schema, partition specs, and manifest lists.

ACID compliance is a set of properties—Atomicity, Consistency, Isolation, and Durability—that guarantee database transactions are processed reliably. Apache Iceberg implements these properties for data lake tables, a critical advancement over raw files.

• Atomicity: Table updates (INSERT, DELETE, MERGE) are committed entirely or not at all.
• Isolation: Readers see a consistent snapshot of the table, unaffected by concurrent writes.
• Durability: Once a commit succeeds, it is persisted and cannot be lost.

Data mesh is a decentralized data architecture that organizes data ownership around business domains, treating data as a product. Apache Iceberg is a key enabling technology for implementing the data product concept.

• Product Interface: Iceberg provides a stable, well-defined table interface (schema, partitions, snapshots) that domain teams can own and manage.
• Interoperability: Its open format allows different domains and consumption tools (Spark, Trino, Flink) to interoperate without proprietary lock-in, supporting the federated governance model of a data mesh.