Skip to content
Source: InfoQ

🧊Apache Iceberg V4 Redesigns Table Metadata for Commits Every Second

Your Iceberg commits could go from minutes to seconds

TL;DR

Apache Iceberg V4 proposes a major redesign of table metadata, including a new metadata tree and more efficient commits. Commits could drop from the current two-minute recommendation to every second or less. If you run large analytical tables, this changes how often you can write.

Apache Iceberg V4 is a rewrite, not a patch. The proposal swaps out the metadata layout so the root of the metadata tree can point directly at data files, which cuts the work needed to add a file to a table. The payoff is commits every second, or even sub-second, where the current guidance is at least two minutes between commits. If you run streaming ingestion into Iceberg, or you manage tables that have outgrown their metadata, this is the part to watch. Frequent writes have been a pain point at scale, and V4 targets exactly that. It also adds typed column statistics in place of V3's binary stats, plus relative file paths that make tables easier to move around. V4 also expands the format into new territory: expression-based default values, check, unique and primary key constraints, and file and vector data types for AI and search workloads. Constraint validation status will be tracked so engines know when to re-check. The catch: V4 metadata will not be backward compatible with older readers, so plan your engine upgrades before you flip anything.

Apache Iceberg V4 Redesigns Table Metadata for Commits Every Second — InfoQ

Key Points

1

The new metadata tree lets the root point directly to data files, simplifying how a file gets added and making frequent commits practical for large tables.

Keep reading — it's free

Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.

Also get
or

Already a member? Sign in

Apache Icebergtable formatsdata lakehousemetadatastreaming

Comments

Subscribe to join the conversation...

Be the first to comment