🧊Apache Iceberg V4 Redesigns Table Metadata for Commits Every Second
Your Iceberg commits could go from minutes to seconds
TL;DR
Apache Iceberg V4 proposes a major redesign of table metadata, including a new metadata tree and more efficient commits. Commits could drop from the current two-minute recommendation to every second or less. If you run large analytical tables, this changes how often you can write.
Apache Iceberg V4 is a rewrite, not a patch. The proposal swaps out the metadata layout so the root of the metadata tree can point directly at data files, which cuts the work needed to add a file to a table. The payoff is commits every second, or even sub-second, where the current guidance is at least two minutes between commits. If you run streaming ingestion into Iceberg, or you manage tables that have outgrown their metadata, this is the part to watch. Frequent writes have been a pain point at scale, and V4 targets exactly that. It also adds typed column statistics in place of V3's binary stats, plus relative file paths that make tables easier to move around. V4 also expands the format into new territory: expression-based default values, check, unique and primary key constraints, and file and vector data types for AI and search workloads. Constraint validation status will be tracked so engines know when to re-check. The catch: V4 metadata will not be backward compatible with older readers, so plan your engine upgrades before you flip anything.

Key Points
The new metadata tree lets the root point directly to data files, simplifying how a file gets added and making frequent commits practical for large tables.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment