What Is Apache Hudi?

Apache Hudi is an open source data lake table format built specifically around fast, incremental writes, upserts, and deletes at scale, along with built-in tooling for ingesting continuously changing source data, like a stream of database changes, directly into a lake table that stays queryable the whole time it is being updated. That combination of speed and built-in ingestion support is what distinguishes Hudi's design goals from the other major open table formats from the very start. A stream of database changes is the clearest example, though the same mechanics apply to any source that produces a steady flow of individual record-level changes rather than one big periodic batch.

It exists because a large share of real-world data lake workloads are not clean, append-only batches, they are streams of inserts, updates, and deletes flowing continuously from operational databases through change data capture. Older lake approaches were built around writing large batches occasionally, not around efficiently applying a constant stream of individual record updates, and Hudi was originally built at Uber specifically to handle exactly that kind of continuously changing data at real scale. Hudi was originally built at Uber specifically to handle exactly that kind of continuously changing data at real scale, and the problem it was solving there has only become more common as more companies lean on change data capture for ingestion.

What separates Hudi from the naive approach of just rewriting entire partitions to apply a handful of updates is a combination of indexing, which locates which existing files actually contain the rows being changed, and a choice of file layout strategy, copy-on-write, which rewrites affected files immediately at write time, or merge-on-read, which logs changes separately and merges them in at read time, so upserts and deletes can be applied without rewriting the whole table every time. Both approaches exist because no single tradeoff between write speed and read speed fits every workload equally well.

By 2026 Hudi is widely used in organizations with heavy change data capture ingestion needs and streaming pipelines, particularly wherever near-real-time freshness on frequently updated data actually matters. Alongside Iceberg and Delta Lake it is one of the three established open lakehouse table formats, though most surveys of the space still show it with a smaller ecosystem and community footprint than the other two. Alongside Iceberg and Delta Lake it is one of the three established open lakehouse table formats, though most surveys of the space still show it with a smaller ecosystem and community footprint than the other two by most measures available.

This page covers how Hudi's indexing and file layout actually work, how it compares to Delta Lake, how it differs from a change data capture tool specifically, and where its upsert-focused design earns its keep versus where a simpler format serves better. The idea worth keeping is that Hudi's whole reason for existing is upsert performance at scale, and that is the right lens for judging whether it fits a given workload. Judging Hudi against that specific bar, rather than against a generic checklist of lakehouse features, is the fastest way to tell whether it actually fits a given job.

• Apache Hudi is an open source table format built specifically for fast, incremental upserts and deletes at scale, particularly from continuous change data capture streams.
• It exists because much of real-world lake data arrives as a continuous stream of updates and deletes, not clean append-only batches, which older lake approaches handled poorly.
• What separates it from naive rewriting is a combination of indexing and a choice between copy-on-write and merge-on-read storage layouts to apply changes efficiently.
• By 2026 it is one of the three established open lakehouse table formats, favored where near-real-time freshness on frequently updated data genuinely matters.
• Its core reason for existing is upsert performance at scale, which is the right standard for judging whether it fits a given workload.

Every Hudi table maintains a timeline, an ordered log of every action taken on it, commits, compactions, cleaning operations, that lets readers and writers agree consistently on the current state of the table without needing to guess at what has happened based on the files present in storage. That timeline is what makes concurrent readers and writers able to agree on a consistent view even while changes are actively being applied underneath them. Older approaches that inferred table state purely from a directory listing had no equivalent mechanism to fall back on.

Incoming writes pass through an indexing layer that maps each record's key to the specific file group it belongs to, so an upsert can go directly to the relevant files instead of scanning the entire table to figure out where a given row already lives, which is what makes updates at scale actually practical. Without that indexing layer, every upsert would have to scan the entire table just to find the handful of rows that actually need to change. Locating the right file group directly is what makes the whole approach practical at real scale.

Two storage layouts handle the resulting merge differently. Copy-on-write rewrites the affected data files immediately at write time, which keeps reads simple and fast since there is nothing left to merge later. Merge-on-read instead writes changes to a separate delta log and merges them with the base file only when a read happens, or during a background compaction, trading some write speed for extra read complexity. Choosing between the two is really a choice about which side of the pipeline you want to bear the extra cost, the write path or the read path.

Background housekeeping, compacting merge-on-read logs, cleaning up old file versions, and clustering to reorganize small files, keeps a Hudi table performing well over time, and it genuinely needs to run on a regular schedule, since a table left without this maintenance degrades noticeably faster than some other formats tolerate similar neglect. Teams that treat this housekeeping as optional tend to find out the hard way, usually through a slow and steadily worsening query rather than a dramatic failure. Set the schedule once, and revisit it as the write volume grows rather than assuming the original cadence will hold forever.

Both Hudi and Delta Lake are open table formats offering ACID transactions and upsert support on lake data, and by 2026 their core transactional capabilities look fairly close on paper, which is part of why the choice between them often comes down to workload shape rather than a clear feature gap. That similarity on paper is exactly why the real decision usually comes down to the shape of the workload rather than a missing feature on either side. Neither vendor treats the comparison as settled, and both keep shipping features that narrow whatever gap remains.

Hudi's explicit choice between copy-on-write and merge-on-read gives more direct control over the write-speed versus read-speed tradeoff for upsert-heavy workloads specifically, which was Hudi's original specialty from day one, while Delta Lake's simpler single storage model tends to be an easier default when a workload is more batch-oriented and less dominated by continuous upserts. Teams running heavy CDC ingestion with frequent record-level changes tend to notice that control the most, since it maps directly onto the problem they actually have.

Hudi also ships more built-in ingestion tooling out of the box, including support for pulling in change data capture streams directly, while Delta Lake workloads more often lean on separate tools, Spark Structured Streaming or a dedicated CDC product, to handle that part of the pipeline before data ever reaches the table format itself. That built-in tooling can meaningfully shorten the time it takes to stand up a working ingestion pipeline from a cold start. Delta Lake users typically wire that ingestion step together themselves using separate, well-established tooling instead.

Community size and hiring pool tend to favor Delta Lake and Iceberg somewhat over Hudi, which is a real, practical factor to weigh even in situations where Hudi is technically the better fit for a genuinely upsert-heavy workload, since finding people who already know the format well can be harder. That factor alone pushes some teams toward the format with the larger hiring pool even when Hudi would technically serve the workload better. Weigh the hiring question honestly before assuming pure technical fit will decide the outcome on its own.

A change data capture tool's job ends at capturing database changes and streaming them somewhere else. It does not own the destination table format or decide how those changes get organized once they land, that part of the job belongs to whatever system receives the stream. Where that stream actually goes, and how it gets organized once it lands, is an entirely separate concern from capturing it in the first place. Conflating the two roles is an easy mistake for anyone new to the pipeline to make.

Hudi is the landing side of that story: the table format and write engine that takes a stream of changes, often produced by a separate CDC tool, and efficiently applies them as upserts and deletes into a queryable lake table, using its own indexing and file management to do so. That combination of capture and landing, handled by two different pieces of technology, is a common pattern across most real-world CDC pipelines built in practice. That handoff between capture and landing is worth diagramming explicitly when onboarding anyone new to the architecture.

People sometimes describe using Hudi for CDC as though Hudi does the capturing itself, but that framing misses what is actually happening. Hudi is what makes the destination table handle the resulting inserts, updates, and deletes efficiently once a separate tool has already captured and delivered them. Getting this framing right avoids a fair amount of confusion when a pipeline needs debugging and nobody is sure which layer actually owns the problem. A clear mental model here saves real debugging time later.

Keeping this distinction straight matters when designing a pipeline, since a team still needs a CDC tool to capture changes at the source even when Hudi is the format receiving and applying them, and assuming Hudi alone covers the whole pipeline leaves a real gap in the architecture. Teams that skip a proper CDC tool and expect Hudi to somehow cover that ground end up with a pipeline that has a real hole in it. Plan for both pieces from the start rather than discovering the gap partway through a build.

Hudi fits well for pipelines ingesting continuous change streams from operational databases where records get updated or deleted frequently, and where near-real-time freshness in the resulting lake table genuinely matters to whoever is querying it downstream. That kind of workload is precisely what the format's timeline, indexing, and layout choices were designed around from day one. Near-real-time freshness on exactly that kind of data is the scenario Hudi was purpose-built to serve well. Plenty of production pipelines fit this description almost exactly.

It also fits well for workloads that need fine-grained upsert performance specifically, rather than just periodic batch appends, since that is the exact problem the format's indexing and layout choices were built to solve. Workloads without that specific need generally have simpler, lower-maintenance options available to them. Fine-grained upsert performance is the whole reason the format exists in the first place. Everything else about the format follows from that one specialty. Most teams recognize this need well before they go looking for a table format to solve it.

It fits poorly for simple, append-only analytic workloads, clickstream logs, event data that is never updated after it lands, where the upsert machinery adds real operational complexity without any meaningful benefit over a simpler format built for straightforward appends. Reaching for Hudi there adds real operational weight to a problem that never asked for it. A format built purely for appends will usually outperform Hudi there with a fraction of the operational overhead. Simplicity wins comfortably in that scenario. Reach for it only when the workload actually needs it.

It also fits poorly for teams without in-house expertise to manage the compaction and cleaning housekeeping a Hudi table needs, since neglecting that maintenance degrades performance more noticeably than with some other formats, and for organizations that weigh ecosystem size heavily, since Hudi currently has the smallest community footprint of the three major open table formats. Weigh both factors honestly rather than choosing purely on technical merit and being surprised by the maintenance burden or hiring gap later. Both factors deserve real weight in the decision.

Choose between copy-on-write and merge-on-read deliberately, based on your actual read-versus-write pattern, rather than defaulting to whichever one a tutorial happened to use. A write-heavy workload with light reads wants a different tradeoff than a read-heavy workload with occasional updates. Getting this choice wrong early tends to show up later as a performance problem that looks unrelated to the original decision. Revisit the choice if the workload's shape changes meaningfully over time rather than treating it as permanent. A quick review now beats a slow surprise later.

Schedule compaction and cleaning as a genuine, first-class part of the pipeline rather than an afterthought bolted on later. A merge-on-read table with neglected compaction slows down more and more over time, and by the time the slowdown is obvious to end users, the backlog of unmerged changes is already substantial. Treat this scheduling the same way you would treat any other production job that the rest of the pipeline quietly depends on. A missed cycle compounds quickly into a much larger backlog than it looks like at first.

Pick record keys and partitioning that actually match how your updates arrive in practice, since Hudi's indexing performance depends heavily on how efficiently it can locate the right file group for a given key, and a mismatch there quietly undermines the whole point of using the format. A mismatch here is one of the more common, and more preventable, reasons a Hudi table underperforms despite the format itself being sound. Test this assumption rather than guessing at it up front.

Use Hudi's incremental query capability to build downstream pipelines that only process what actually changed since the last run, instead of re-reading the entire table every time, which is both faster and closer to how the format was designed to be used in the first place. That efficiency gain compounds as the table and its change volume grow larger over time. Reprocessing everything from scratch on every run wastes compute that incremental processing was specifically built to avoid. That efficiency matters most once tables grow large enough that a full scan becomes genuinely expensive.

Budget real, ongoing operational attention for a Hudi deployment. It rewards teams willing to tune indexing, layout choice, and maintenance schedules deliberately, and it punishes teams that set it up once and then treat it as something that should just run itself indefinitely without anyone watching. Teams that give it that attention tend to be glad they chose it; teams that do not tend to regret the choice fairly quickly. Ongoing attention is the price of the format's real strengths.

• Choose copy-on-write or merge-on-read based on your actual read-versus-write pattern rather than defaulting to one without evaluating the tradeoff.
• Schedule compaction and cleaning as a first-class, recurring part of the pipeline rather than something addressed only after performance visibly degrades.
• Select record keys and partitioning that match how updates actually arrive, since indexing performance depends heavily on locating the right file group efficiently.
• Use incremental query capability to process only what changed since the last run, instead of re-reading the entire table on every pipeline execution.
• Budget ongoing operational attention for tuning and monitoring, since Hudi rewards active maintenance and degrades faster than some formats when neglected.

• Apache Hudi is not a change data capture tool itself; it is the table format that efficiently applies changes a separate CDC tool has already captured and delivered.
• Hudi is not a drop-in choice for every lake workload; simple append-only data gains little from its upsert machinery and pays real complexity cost for no benefit.
• Merge-on-read tables do not maintain themselves; neglected compaction causes reads to slow down progressively as unmerged changes accumulate.
• Hudi and Delta Lake are not simply interchangeable; the right choice depends heavily on whether a workload is genuinely upsert-heavy or more batch-oriented.
• A smaller community footprint is not a minor detail; it is a real practical factor affecting hiring and available expertise when adopting Hudi.