Lakehouse Table Formats: Apache Iceberg, Apache Hudi, and Delta Lake Compared - Data Engineering Hub

← Back to Home Lakehouse Table Formats: Apache Iceberg, Apache Hudi, and Delta Lake Compared Table of Contents Introduction CDC Approaches Debezium Kafka Connect Exactly-Once Schema Evolution Monitoring Operational Best Practices Comparison Recommendations Conclusion The Lakehouse Revolution Lakehouse architectures combine the best of data lakes (open formats, cheap storage) and data warehouses (ACID, performance). The key enabler is the open table format. Apache Iceberg Born at Netflix, Iceberg is designed for huge analytic tables. Core Architecture Catalog -> Metadata Files -> Manifest Lists -> Manifest Files -> Data Files (Parquet/ORC/Avro) Key Features Hidden Partitioning: Partition spec is metadata, not physical layout Partition Evolution: Change partitioning without rewriting data Time Travel: Query historical snapshots by timestamp or snapshot ID Schema Evolution: Safe add/drop/rename/reorder columns Row-Level Operations: MERGE, DELETE, UPDATE with positional delete files Catalog Options Hive Metastore AWS Glue Nessie (Git-like branching) REST Catalog (Open Table Format spec) Spark Example // Write df.writeTo("catalog.db.table") .tableProperty("write.format.default", "parquet") .partitionedBy(col("event_date"), bucket(16, col("user_id"))) .createOrReplace() // Time travel spark.read.format("iceberg") .option("snapshot-id", 12345) .load("catalog.db.table") // Partition evolution (metadata only!) ALTER TABLE catalog.db.table WRITE PARTITIONED BY (day(event_date), bucket(16, user_id)) Apache Hudi Born at Uber, Hudi optimizes for upsert-heavy workloads and incremental processing. Table Types Copy-on-Write (CoW): Full file rewrite on update. Best for read-heavy. Merge-on-Read (MoR): Columnar base + row-based log files. Best for write-heavy. Key Features Incremental Queries: Read only changed records since last checkpoint Clustering: Automatic file sizing and layout optimization Compaction: Merge log files into base files (MoR) Indexing: Bloom filters, record-level indexes Write Configuration hoodie.datasource.write.table.type=MERGE_ON_READ hoodie.datasource.write.operation=upsert hoodie.datasource.write.precombine.field=updated_at hoodie.datasource.write.recordkey.field=id hoodie.datasource.write.partitionpath.field=event_date hoodie.compact.inline=true hoodie.compact.inline.max.delta.commits=4 hoodie.clustering.inline=true hoodie.clustering.inline.max.commits=4 Delta Lake Born at Databricks, Delta Lake emphasizes simplicity and Spark integration. Key Features ACID Transactions: Serializable isolation via optimistic concurrency Time Travel: VERSION AS OF, TIMESTAMP AS OF Schema Enforcement/Evolution: MergeSchema, OverwriteSchema DML Support: MERGE, DELETE, UPDATE, INSERT Z-Ordering: Multi-dimensional clustering for query performance Delta Lake Operations // Write with schema evolution df.write.format("delta") .option("mergeSchema", "true") .mode("append") .save("/mnt/delta/events") // Time travel spark.read.format("delta") .option("versionAsOf", 5) .load("/mnt/delta/events") // Optimize with Z-ordering OPTIMIZE delta./mnt/delta/events ZORDER BY (user_id, event_type) // Vacuum old files (default 7 days retention) VACUUM delta./mnt/delta/events RETAIN 168 HOURS Detailed Comparison Feature Iceberg Hudi Delta Lake Partition Evolution Native No No Hidden Partitioning Yes No No Row-Level Deletes Positional delete files Merge-on-Read Delete vectors (3.0+) Incremental Queries Changelog views Native (primary feature) Change Data Feed Clustering/Compaction Rewrite files Built-in clustering OPTIMIZE + ZORDER Schema Evolution Full (add/drop/rename/reorder) Add/drop/rename Add/drop/merge/overwrite Catalog Options Many (Hive, Glue, Nessie, REST) Hive, Glue Hive, Glue, Unity Catalog Streaming Writes Flink, Spark Structured Streaming Flink, Spark Structured Streaming Engine Support Spark, Flink, Trino, Presto, Dremio Spark, Flink, Presto, Trino Spark, Flink, Presto, Trino Cloud Vendor Backing AWS, Tabular, Snowflake Onehouse, AWS Databricks, Microsoft, Linux Foundation Performance Characteristics Read Performance Iceberg: Best for scan-heavy workloads with partition pruning Hudi MoR: Good with proper compaction; log files can slow scans Delta: Excellent with Z-ordering for multi-column filters Write Performance Iceberg: Consistent; manifest list updates are lightweight Hudi CoW: Slower (full rewrite); MoR faster for upserts Delta: Fast appends; MERGE performance depends on join strategy Storage Efficiency Iceberg: Best with hidden partitioning (no physical partition directories) Hudi: MoR can have temporary storage overhead during compaction Delta: Good; delete vectors (3.0+) improve delete efficiency Migration Strategies Hive to Iceberg // In-place migration (metadata only) CALL catalog.system.migrate('hive_db.table') // Or create Iceberg table from Hive CREATE TABLE iceberg_db.table LIKE hive_db.table TBLPROPERTIES ('format-version'='2') Delta to Iceberg // Using Delta's manifest files CALL catalog.system.register_table( table => 'iceberg_db.table', metadata_file => 's3://bucket/delta-table/_delta_log/00000000000000000000.checkpoint.parquet' ) Recommendations by Use Case Use Case Recommended Format Reason Partition evolution needed Iceberg Only format with native partition evolution High-frequency upserts (CDC) Hudi MoR Optimized for merge-on-read with clustering Databricks ecosystem Delta Lake Native integration, Unity Catalog Multi-engine (Spark + Flink + Trino) Iceberg Broadest engine support, REST catalog Time travel / audit compliance All three All support time travel; Iceberg has most flexible snapshot model Incremental processing / Change capture Hudi Incremental queries are a first-class feature The Converging Future The Open Table Format specification (Iceberg-led) and Apache XTable (formerly Hudi's multi-format support) are driving convergence. Soon you may query any format with any engine seamlessly. Related Articles Batch Processing at Scale Streaming Data Pipelines Modern Data Warehousing Data Quality and Testing Data Observability Data Orchestration