What Is Apache Iceberg?

Apache Iceberg is an open table format for large analytic datasets stored in cloud object storage or a data lake. It takes files of data, almost always Parquet, and wraps them in a metadata layer that gives them the transactional guarantees, schema management, and query performance you would expect from a proper database table, while letting many different engines, Spark, Trino, Flink, Snowflake, and others, read and write the same underlying data reliably. That combination of guarantees and openness is what most people actually mean when they call something a lakehouse table format in 2026.

It exists because building a data lake directly on raw files in folders came with real, well-documented problems. Concurrent writers could corrupt what a reader saw mid-write, planning a query over a huge folder meant listing potentially millions of files, which got slow fast, schema changes were painful and often silently broke downstream consumers, and there was no reliable way to look at an earlier version of a table or roll back a bad write. Iceberg was built specifically to fix those problems without forcing anyone to move their data into a single proprietary warehouse.

What separates Iceberg from the older, Hive-style convention of tracking a table by its directory structure is that Iceberg keeps an explicit metadata layer of manifests and snapshots recording exactly which files make up a given version of the table. That explicit tracking is what enables atomic commits, fast partition pruning without listing directories, schema evolution that does not require rewriting existing data, and time travel to any prior snapshot, none of which a plain folder of files can offer on its own. None of that is possible when a table's boundaries are inferred from whatever files happen to sit in a folder at query time.

By 2026 Iceberg has become one of two or three dominant open table formats, with native read and write support built into most major cloud data platforms and warehouses. It is the common choice for organizations that specifically want an open, engine-agnostic lakehouse rather than being locked into one vendor's proprietary storage layer, and that neutrality has been a real part of its appeal as more engines added first-class support. That neutrality has been a real part of its appeal as more engines added first-class support over the past several years.

This page covers the mechanics of how Iceberg's metadata layer actually works, how it compares to Delta Lake, how it differs from the older Hive table convention it largely replaces, and where it earns its keep versus where it is more machinery than a workload actually needs. The idea worth holding onto is that Iceberg's real contribution is making a pile of files behave like a proper table, with the vendor neutrality that lets many different engines share it without a fight. Everything that follows is really an elaboration on that one basic idea.

• Apache Iceberg is an open table format that adds transactional guarantees, schema management, and multi-engine support to files, usually Parquet, in a data lake.
• It exists to fix real problems in raw file-based data lakes: corrupted concurrent writes, slow query planning at scale, painful schema changes, and no reliable time travel.
• What separates it from the older Hive convention is an explicit metadata layer of manifests and snapshots that tracks exactly which files make up each table version.
• By 2026 it is one of the dominant open table formats, widely chosen by organizations that want an engine-agnostic lakehouse rather than a single vendor's proprietary format.
• Its core contribution is making a set of files behave like a proper database table while staying open enough for many different engines to share it.

Every Iceberg table is organized around a metadata tree, starting with a table metadata file that points to the current snapshot, which itself references a manifest list, which in turn references one or more manifest files listing the actual data files along with statistics like minimum and maximum values and row counts for each one. That layered structure is what lets an engine plan a query without ever touching the raw data files directly. Statistics recorded at each level let an engine skip whole branches of that tree without opening a single data file.

When a write happens, an insert, update, or delete, Iceberg creates new data files and new metadata entries, then atomically swaps the table's pointer to a new snapshot once everything is written. Readers never see a half-finished write, because they are always looking at a complete, consistent snapshot rather than whatever files happen to exist in a folder at a given moment. That atomicity is the property that makes concurrent writers safe in a way a plain folder of files never could be.

That same snapshot structure is what makes time travel and rollback possible. Querying a table as of an earlier snapshot, or rolling back to one after a bad write corrupts the current version, is simply a matter of pointing at an older metadata entry rather than reconstructing anything from scratch, which is a meaningfully different experience from trying to recover a plain folder of overwritten files. Recovering from a bad write used to mean restoring from a backup; now it usually means pointing at yesterday's snapshot instead.

Query planning also benefits directly from this structure, since an engine can prune irrelevant partitions and files using the statistics already recorded in the manifests, without ever listing a directory. Schema evolution, adding a column, renaming one, or widening a type where it is safe to do so, is handled purely at the metadata level too, so existing data files do not need to be rewritten just because the schema changed. Both capabilities exist specifically because the metadata layer tracks state explicitly instead of leaving an engine to guess.

Iceberg and Delta Lake solve essentially the same core problem, giving transactional, versioned behavior to tables built on lake storage, and by 2026 both are mature, widely used, and have closed a lot of the feature gaps that used to separate them, ACID transactions, time travel, and schema evolution are solid on both sides of the comparison. The real differences by this point show up more in tooling depth and ecosystem fit than in raw capability on paper. Neither vendor is standing still, so any gap tends to close further with each release.

Iceberg was designed from the outset to be engine-agnostic, and that shows in how broadly it is supported across Spark, Trino, Flink, and a range of warehouses and catalogs that were not built by a single vendor. That neutrality has made it a natural default for organizations that specifically do not want to be tied to one company's ecosystem. Organizations that expect to add engines over time tend to weigh that history heavily when making the initial choice. That history is hard to erase even after the raw feature comparison stops favoring either side clearly.

Delta Lake grew up tightly coupled to Databricks and Spark, and while support outside that world has genuinely expanded, the deepest, most polished integration still tends to live inside the Databricks ecosystem specifically, with features like automated file management that fit together more smoothly there than they do in a more mixed environment spanning several vendors. Teams outside that ecosystem generally get a solid experience too, just not quite the same polish. The gap has narrowed but has not fully disappeared as of 2026.

In practice, the choice between the two often comes down less to a clear technical winner and more to which ecosystem and catalog you are already committed to. Teams heavily invested in Databricks tend to lean Delta Lake for the tooling depth, while teams spanning multiple engines or vendors more often lean toward Iceberg for the neutrality. Neither choice is wrong on technical grounds alone; the fit is mostly about where you already are. Asking which engines you will actually run in three years is a more useful question than asking which format is theoretically better today.

A Hive-style table defines what data belongs to it essentially by convention: everything sitting in a given directory, and its partition subdirectories, counts as part of the table. That convention worked fine at modest scale, but it means an engine has to list potentially enormous numbers of files just to plan a query, and there is no atomic way to change many files at once. That listing cost grows roughly with the size of the table, which is exactly the wrong direction as data volume increases over time.

Schema changes on a Hive table are also risky in a way that has burned plenty of teams. Renaming or reordering a column can silently misalign data that was written under the old schema, since the table's structure is inferred from the files rather than tracked explicitly, and there is no built-in mechanism to catch that kind of mismatch before it causes wrong results. Plenty of production incidents trace back to exactly this kind of silent schema drift going unnoticed for months.

Iceberg replaces the directory-listing convention entirely with an explicit, versioned record of exactly which files make up the table at any given point, tracked in metadata rather than inferred from a filesystem. That single change is what makes atomic commits, safe schema evolution, and fast query planning possible in a way Hive's convention never supported. That single change is really the whole story behind everything else Iceberg is able to offer. Nothing about that tracking depends on trusting a filesystem listing to be complete or current.

The practical result is that migrating from Hive tables to Iceberg is less about changing file formats and more about changing what the table's boundaries actually mean, from an implicit convention that engines have to trust, to an explicit, checkable record that removes a lot of the guesswork a Hive table always required. Teams that have made this move describe it less as a migration and more as finally trusting the table's own bookkeeping. That reframing is subtle but it changes how teams think about risk during the change itself.

Iceberg fits well for large analytic datasets that multiple engines need to read from and write to, especially when avoiding lock-in to a single vendor's proprietary format matters to the organization. It also fits well anywhere transactional correctness on lake-stored data genuinely matters, concurrent writers, frequent deletes or updates, without wanting to move everything into a traditional warehouse. Avoiding that lock-in is often reason enough on its own for a platform team to standardize on Iceberg going forward. That combination of scale and multi-engine access is really the sweet spot the format was designed around.

It fits well for organizations running a genuinely heterogeneous stack, several query engines, maybe more than one cloud, where a shared, open table format avoids duplicating data or building brittle bridges between systems that each expect their own proprietary format. A shared format also means less duplicated storage and fewer brittle, one-off bridges between systems to maintain over time. Avoiding proprietary lock-in tends to matter more the larger and more heterogeneous the stack becomes over time. Fewer bridges also means fewer places for something to quietly break at three in the morning.

It fits poorly for small datasets where the metadata overhead of manifests and snapshots has no real payoff compared to just working with a handful of Parquet files directly. The machinery Iceberg provides is solving problems that only show up at real scale. Adopting Iceberg there is a bit like buying enterprise software to manage a grocery list. The juice rarely justifies the squeeze at that scale. Simplicity wins comfortably at that end of the scale. Save the format's real strengths for the workloads that actually need them.

It also fits poorly as a stand-in for an operational database. Iceberg is built for analytic batch and streaming patterns, not for high-frequency, low-latency single-row updates the way a transactional database handles them, and trying to use it that way will disappoint anyone expecting database-grade latency. Reaching for Iceberg to replace a transactional database is a mismatch that tends to surface quickly and painfully. Anyone expecting single-row, millisecond latency from a lake table format will end up disappointed no matter how it is configured.

Choose a catalog deliberately, the component that tracks which snapshot counts as current for a table, since the catalog choice affects which engines can reliably read and write the table and how concurrent writes get coordinated. This decision is easy to treat as an afterthought and hard to change cleanly once several engines depend on it. Changing catalogs later, once several teams and engines depend on it, is a genuinely disruptive migration to undertake. Getting expert advice on this choice early is worth far more than it costs.

Plan for regular compaction and maintenance from the start rather than after performance already suffers. Every write leaves behind small files and old snapshots, and if nobody cleans them up, query performance degrades and storage costs climb quietly in the background until someone finally notices. The cost of catching this early, through a scheduled job, is far lower than the cost of noticing it during an incident. A quarterly maintenance job is a small price to pay for a table that keeps performing the way it did on day one.

Use Iceberg's schema evolution features instead of manually rewriting data whenever a column needs to be added, renamed, or widened. Falling back to a manual rewrite defeats the entire point of having a metadata layer that was built to handle exactly that kind of change safely. Falling back to a manual rewrite defeats the entire point of having a metadata layer built to handle exactly that kind of change safely and without drama. Treat the metadata layer as the tool it was built to be, not an obstacle to work around.

Think through partitioning strategy deliberately before loading a large amount of data, since a poorly chosen partition scheme is one of the most common reasons an Iceberg table underperforms, even though the format itself is entirely sound. Iceberg gives you the tools; it does not choose good partitions for you automatically. A partitioning scheme that made sense at launch can quietly stop making sense once the table grows by an order of magnitude. Revisiting the scheme as the table grows is a normal part of operating it well, not a sign something went wrong.

Test multi-engine read and write compatibility against your actual stack before committing to it in production, since not every engine implements every Iceberg feature at the same level of maturity, and discovering a gap after you are already relying on it in production is a far more expensive way to find out. A short compatibility test before launch is far cheaper than an outage discovered by a downstream consumer in production. A short list of must-have features checked against your actual engines beats trusting a vendor's compatibility matrix at face value.

• Choose a catalog deliberately, since it determines which engines can reliably read and write the table and how concurrency is coordinated.
• Schedule regular compaction and cleanup of small files and old snapshots, since neglecting this quietly degrades query performance and inflates storage cost.
• Use built-in schema evolution features for column changes instead of manually rewriting data, which defeats the purpose of the metadata layer.
• Design partitioning strategy deliberately before loading significant data, since a poor partition scheme is a common cause of underperformance.
• Test multi-engine compatibility against your actual production stack before relying on it, since feature maturity varies between engines.

• Apache Iceberg is not a database; it is a table format for lake-stored data, built for analytic batch and streaming patterns, not high-frequency single-row transactions.
• Iceberg does not require moving data into one vendor's proprietary warehouse; the whole point of the format is engine-agnostic access to data that stays in open storage.
• Choosing Iceberg does not automatically guarantee good query performance; a poorly chosen partitioning strategy can still make an Iceberg table slow despite the format being sound.
• Iceberg tables do not maintain themselves; without regular compaction and cleanup, small files and old snapshots accumulate and degrade performance over time.
• Not every query engine supports every Iceberg feature equally well; feature maturity varies, and that gap should be tested rather than assumed.