Definition
A Parquet file is a way of saving tabular data to disk where the values are grouped by column instead of by row. Open one up and you will not find row after row of records the way a spreadsheet or a CSV lays things out. Instead, every value from one column sits together, then every value from the next column, and so on. That single choice, storing by column, is the whole reason the format exists and the reason it behaves so differently from the formats most people learn first. It was built specifically for the kind of analytical queries that scan millions or billions of rows but only touch a handful of columns at a time.
The problem Parquet solves is wasted reading. A CSV or a row-oriented database file has to pull an entire row off disk even if your query only needs two of its twenty columns, because the row is the unit that gets stored and read together. At the scale of a data warehouse or a data lake, that waste adds up to real time and real money, since you are paying to move bytes you never use. Parquet flips the storage order so that a query asking for two columns only has to read those two columns, skipping the rest of the file entirely. That is the direct answer to a very old and very expensive problem in analytics.
What separates Parquet from simply writing columns instead of rows is everything wrapped around that basic idea. Each column gets its own compression and encoding chosen for the kind of data in it, so a column of repeated country codes compresses very differently from a column of unique timestamps. The file also stores summary statistics per chunk, like the minimum and maximum value, so an engine can skip whole sections without reading them at all. A naive column-oriented file gets you less reading; a real Parquet file gets you less reading, smaller files, and skipped sections, which together are what make it fast in practice.
By 2026, Parquet is close to the default file format for analytical data sitting in cloud storage. It sits underneath most data lakes, gets written and read by engines like Spark, Trino, DuckDB, and Snowflake's external tables, and forms the storage layer that newer table formats like Iceberg and Delta Lake build on top of. Almost nobody chooses Parquet because it is trendy anymore. Teams choose it because the alternatives are visibly slower and larger at any real scale, and the tooling to read and write it is now everywhere.
This page covers how Parquet actually organizes data internally, how it stacks up against CSV, what separates it from a plain database table, and where it is the wrong tool despite its popularity. The durable idea to hold onto is that Parquet is a storage format optimized for reading a few columns out of a lot of rows, cheaply and repeatedly. Everything else about it, the compression, the metadata, the row groups, exists to serve that one goal.
Key Takeaways
- A Parquet file stores data by column rather than by row, so queries only read the columns they actually need.
- It exists to cut the wasted disk reads and cost that come from scanning full rows in formats like CSV at large scale.
- Its speed comes from a combination of per-column compression, encoding, and embedded statistics that let engines skip data, not just from column order alone.
- By 2026 it is the de facto storage format under most data lakes and underneath newer table formats like Iceberg and Delta Lake.
- The durable idea is that Parquet trades write-time simplicity for cheap, repeated, column-selective reads at scale.
How a Parquet File Works
A Parquet file starts by splitting a table into row groups, chunks of some number of rows, often in the range of tens of thousands to a few hundred thousand depending on how the writer configured it. Within each row group, the data is stored column by column rather than row by row. So instead of one block holding an entire record, you get one block holding every value from column A for that chunk of rows, then a block for column B, and so on. This is the layout that lets an engine jump straight to the columns it needs and ignore the rest.
Each column chunk carries its own encoding, chosen to fit the data. A column with only a few repeated values, like a status flag, might use dictionary encoding, storing the distinct values once and referencing them by a short code. A column of steadily increasing numbers might use delta encoding. On top of the encoding, Parquet applies general-purpose compression, commonly something like Snappy or Zstandard, which shrinks the file further because compressing similar values together works better than compressing a mixed row.
Metadata is where a lot of the real speed comes from and it sits at the end of the file, not scattered through it. That footer records the schema, how many row groups exist, and for each column chunk within each row group, statistics like the minimum, maximum, and count of values. An engine reads the footer first, decides which row groups and which columns are even worth opening based on those statistics, and skips everything else without ever touching the actual data.
Reading a Parquet file, then, is less like scanning a file top to bottom and more like consulting an index and jumping to exactly what is needed. Writing one is more work up front, since the writer has to buffer enough rows to build a row group, choose encodings per column, and compute statistics before anything is flushed to disk. That tradeoff, more effort at write time for much less effort at read time, is deliberate, because analytical workloads read the same data far more often than they write it.
A Parquet File Compared to a CSV File
CSV is about as simple as file formats get: plain text, one row per line, values separated by commas. That simplicity is its whole appeal. You can open a CSV in a text editor, generate one from almost anything, and hand it to almost any tool without worrying about compatibility. Parquet has none of that casual accessibility. It is a binary format with an internal structure that needs a library to read, which is a real cost when you just want to eyeball a small file quickly.
Where the two part ways hard is at scale. A CSV file storing a billion rows has to be read start to finish, row by row, even if a query only cares about one column, and it carries no compression beyond whatever you apply externally. The same data in Parquet is typically a fraction of the size on disk because of column-aware compression, and a query touching one column reads roughly that one column's worth of bytes rather than the whole file. The gap between the two widens as the data gets bigger, not smaller.
CSV also has no real schema. Every value is text until something downstream decides to parse it as a number or a date, and a badly quoted field or a stray comma can silently corrupt a row without anyone noticing until later. Parquet enforces a schema at write time, with actual types for each column, so a lot of the ambiguity and quiet corruption that plagues CSV pipelines simply cannot happen the same way in a Parquet file.
The honest comparison is that CSV wins on universality and on being human-readable, which matters for small files, quick exports, and moving data between systems that do not share a Parquet library. Parquet wins decisively once the data is large and the workload is analytical, reading a subset of columns across a lot of rows, repeatedly. Most teams end up using both, CSV for small interchange and manual inspection, Parquet for anything that actually needs to be queried at scale.
What Makes a Parquet File Different From a Database Table
A database table lives inside a database engine, which manages storage, indexing, transactions, and concurrent access to it as a single system you interact with through SQL. A Parquet file is just a file sitting in storage, with no engine attached to it and no built-in notion of transactions or concurrent writers. You can put a Parquet file on cheap object storage and leave it there; you cannot do that with a live table inside a running database.
That difference in ownership matters more than it sounds. A database table is mutable in the sense that the engine handles updates, deletes, and locking for you, and it typically enforces constraints and indexes that speed up specific lookups. A Parquet file is, on its own, closer to a snapshot: writing new data usually means writing a new file rather than editing rows in place, and there is no index beyond the coarse statistics baked into its metadata.
This is exactly the gap that table formats like Iceberg, Delta Lake, and Hudi were built to close. They sit on top of a collection of Parquet files and add the things a database table normally provides: a transaction log, the ability to update or delete specific rows, schema evolution tracked over time, and a consistent view for multiple readers and writers. Underneath most of them, the actual bytes are still Parquet, arranged and tracked with a layer of bookkeeping.
So the practical distinction is that a database table is a live, managed thing you query directly, while a Parquet file is a storage format that becomes table-like only when something else, a query engine or a table format, wraps structure around a set of files. Confusing the two leads to real mistakes, like expecting row-level updates on raw Parquet files the way you would get from a database, and being surprised when that requires rewriting whole files instead.
Where Parquet Files Fit and Where They Do Not
Parquet fits naturally in data lakes and lakehouses, where large volumes of historical data sit in cheap storage and get queried by analytical engines that only need a subset of columns for any given question. Reporting, dashboarding, and ad hoc analysis over years of event or transaction data are the textbook use case, since those workloads read a lot of rows but rarely need every column at once. It also fits well as the storage layer under modern table formats, which is exactly how most large-scale lakehouses are built today.
It also fits well for archiving and for moving data between systems that both understand it, since the compression keeps storage costs down and the embedded schema avoids the guesswork that plain text formats leave behind. Machine learning pipelines that need to repeatedly scan large feature sets for training also benefit, since reading a handful of feature columns out of a much wider table is precisely what the format is optimized for.
Parquet fits poorly for workloads that need to read or write single rows at a time. Looking up one customer's record by ID, updating a single field, or handling high-frequency transactional writes are all things a row-oriented database does far more efficiently, because Parquet's column layout means touching one row still involves work across every column chunk it belongs to, and files are usually written once rather than edited in place. Using Parquet as the backing store for an application's live transactional data is a mismatch that tends to surface as poor performance rather than an outright failure.
It also fits poorly for very small datasets, where the overhead of the format, the metadata, the row group structure, buys you nothing and a CSV or a simple in-memory structure would do the job with less fuss. The honest rule is that Parquet earns its complexity at scale and for analytical access patterns, and it is overkill, sometimes actively unhelpful, anywhere those two conditions are not both true.
How to Use Parquet Files Well
Pay attention to row group size before assuming Parquet will be fast by default. Row groups that are too small mean the metadata overhead and the loss of compression efficiency start to outweigh the benefits, while row groups that are too large can hurt parallelism because an engine cannot split work below that boundary. Most write tools have sensible defaults, but if you are tuning a pipeline that writes a lot of Parquet, it is worth checking that the row group size actually matches your typical query pattern.
Choose column order and partitioning with your actual queries in mind rather than the order columns happened to arrive in. Partitioning a large dataset by a column you frequently filter on, like a date, lets engines skip entire files rather than just skipping columns within a file, which is a much bigger win than column pruning alone. This only pays off if the partitioning matches how the data is actually queried, so it is worth checking real query patterns before deciding.
Avoid writing a huge number of tiny Parquet files, which is a common outcome of streaming or frequent small batch writes. Lots of small files mean an engine spends more time opening files and reading footers relative to the actual data it processes, which can slow things down more than the column-oriented format speeds them up. Compacting small files into fewer, larger ones on a schedule is a routine maintenance task in most Parquet-based pipelines, not an optional nice-to-have.
Keep schema evolution deliberate. Parquet supports adding columns and some type changes over time, but different files in the same dataset can end up with slightly different schemas if writers are not coordinated, and that mismatch can quietly break downstream readers that expect consistency. Using a table format like Iceberg or Delta Lake on top of raw Parquet handles a lot of this for you, which is a large part of why those formats have become popular rather than working with bare Parquet files directly.
Match your compression codec to your priorities rather than accepting whatever default a tool ships with. Some codecs compress harder but are slower to read, others are faster to decompress but leave more bytes on disk, and the right choice depends on whether you are optimizing for storage cost or query latency. It is a small setting, but at real data volumes the difference between codecs shows up clearly in both storage bills and query times.
Best Practices
- Size row groups to match typical query parallelism instead of leaving defaults unexamined at large scale.
- Partition files by columns you actually filter on so engines can skip whole files, not just columns.
- Compact small files produced by streaming or frequent writes on a regular schedule to avoid metadata overhead.
- Coordinate schema changes across writers or use a table format on top of Parquet to avoid silent mismatches.
- Pick a compression codec based on whether storage cost or read speed matters more for your workload.
Common Misconceptions
- A Parquet file is not a database; it has no built-in transactions, indexing engine, or concurrent write handling on its own.
- Parquet is not just columns instead of rows; the compression, encoding, and embedded statistics are what actually make it fast.
- Parquet files are not efficient for single-row lookups or updates, which row-oriented systems still handle much better.
- A Parquet file is not human-readable like a CSV; it requires a library or tool to open and inspect.
- Using Parquet does not automatically give you schema evolution or row-level updates; that usually requires a table format built on top of it.
