Writing · Lakehouse internals
The encoder is the read lever, not the table format.
I went into this expecting to find a read-speed difference between Iceberg and DuckLake, because that's the comparison everyone reaches for when they pick a table format, and I'd seen Iceberg come back slower in my own earlier runs. What I found instead is that the table format was never the thing I was measuring. The lever on read speed is the Parquet writer that produced the files, and once I took the writer out of the comparison by registering the byte-identical bytes into both catalogs, the two formats read the same. The same thing turned out to be true of file size, where two writers handed the identical rows and the identical codec still disagreed by almost a factor. So the encoder governs both the read and the bytes on disk, and the format is the catalog around them.
What I thought I was measuring
Is Iceberg slower to read than DuckLake?
The two table formats sit at different points on the spectrum, and the way you usually decide between them is by reputation. Iceberg is the heavier specification with the broad catalog ecosystem and the manifest-based metadata layer, while DuckLake keeps its table metadata in a SQL database and aims to be the lighter, lower-ceremony option. When someone asks me which one reads security telemetry faster, the starting position was that I didn't know, so I set up a scale ladder to find out. I generated synthetic network-connection events and queried them through DuckDB against an Iceberg table and against a DuckLake table at one million, ten million, a hundred million, and a billion rows.
On the first pass Iceberg looked slower, and not by a trivial amount. Across the query set it came back somewhere between about 1.1 times and 1.55 times slower than DuckLake on the same logical data, which is the kind of gap that would feel decisive if you stopped there and wrote it up. If I'd published that table it would have read as a clean format-versus-format result, with DuckLake winning the read path. A practitioner deciding between the two on read latency would have taken Iceberg off the shortlist on the strength of it. The number was real in the sense that the clock genuinely measured it. It just wasn't measuring what the headline would have said it was measuring.
What stopped me from writing it up that way is that the two setups weren't reading the same files. Each format had written its own Parquet through its own path, so when I compared read latencies I was comparing two different sets of bytes that happened to encode the same rows. Any difference between them could just as easily live in the bytes as in the format's read machinery. Before I could say anything about the formats I had to make the files identical, which turned out to be the whole story.
Where the gap actually lived
Iceberg defaulted to ZSTD, DuckLake to Snappy.
The first confound was the easy one to name once I looked. The two paths weren't even compressing the data the same way, because pyiceberg's write path defaulted the files to ZSTD while the DuckLake path wrote Snappy. ZSTD trades decompression work for smaller files in a way that shows up directly on a scan-heavy read. So a chunk of the apparent Iceberg slowness was a codec default sitting underneath the format, not a property of the format itself. The moment you say "I measured Iceberg against DuckLake" while one is on ZSTD and the other is on Snappy, you've already mislabeled the result. The format names were on the axis, but the codec was doing the work.
The fix that seems obvious is to set both writers to the same codec and call it controlled, and that's where the second and more stubborn confound surfaced. Matching the codec does not match the bytes, because the two Parquet writers disagree about everything below the codec. On the identical input data, with the codec held equal, PyArrow's writer produced a file of about 193 MB where DuckDB's writer produced about 114 MB for the same rows. The roughly 1.7-times difference in size has nothing to do with compression algorithm and everything to do with encoding decisions. The largest single contributor I could see is dictionary encoding on the high-cardinality columns, where PyArrow makes different choices about when to dictionary-encode and when to fall back. And pyiceberg gives you no per-column control to override it, so you can't simply tell it to encode the way DuckDB does.
Matching the codec didn't close the read gap either, which puts a middle rung between the confounded default-config comparison and the byte-identical one. At a hundred million rows I wrote both formats on ZSTD and then both on Snappy, held the row-group nominally fixed, and read them through the same DuckDB. On the two sustained aggregations steady enough to rank, the subnet rollup and the heavy GROUP BY, both at a coefficient of variation of roughly two to four percent, Iceberg came back at 1.13 times and 1.14 times on ZSTD and at 1.09 times and 1.18 times on Snappy. I'm leaving the byte rollup and the point lookup out of that reading, because the byte rollup at that scale sits in a band noisy enough that a difference below its own variation isn't real, and a point lookup is too small a query to rank a sustained scan on. When I wrote that arm up I called the residual a real format and read-path effect, DuckLake still reading modestly faster after normalization. I'd read it more narrowly now, because matching the codec had left each writer's per-column encoding free, so those files still weren't the same bytes.
Read parity · confounds stripped in turn
That second confound is the one I'd want a benchmark reader to internalize, because "same codec" feels like it should mean "same bytes" and it doesn't. Two correct Parquet writers, handed the identical rows and the identical compression codec, will still emit substantially different files. And a file that's 1.7 times larger has more to read off disk, more to decompress, and a different layout for the scanner to walk. If that difference is allowed to ride along inside a comparison labeled Iceberg-versus-DuckLake, then the comparison is measuring the writers and reporting on the formats, which is exactly the kind of mislabeled result that makes published read benchmarks untrustworthy.
The same writer governs the bytes on disk
"Iceberg is more storage-efficient" was a codec default that flipped.
I'd actually walked into the same trap on storage before I got to read speed, because the first version of this work set out to answer a different question: whether Iceberg or DuckLake stored the synthetic network-connection corpus more cheaply. Storage is a real cost line when you keep security telemetry for a year or more, and the per-partition difference compounds across the whole retention window. The early numbers told the story everyone tells, that the Iceberg table came out meaningfully smaller. That lined up with the conventional read that Iceberg is the more storage-efficient format, the one the mature ecosystem has tuned hardest. The reason I didn't write that down is the same reason the read result fell apart: pyiceberg defaulted its files to ZSTD while the DuckLake path defaulted to Snappy. So I was comparing a ZSTD file against a Snappy file and calling the difference a property of the format, which is a codec comparison wearing a format comparison's clothes.
When I held the codec constant and let each format use its own default writer, the ranking inverted. At the larger scale I re-ran to confirm it, the DuckLake-written files came out the smaller of the two, 1.14 GB against 1.93 GB for the same dataset. That means the format people call more storage-efficient was, on a matched codec, writing the larger files, because PyArrow's writer made encoding choices that left more bytes on disk than DuckDB's did. That reading is no more accurate than the original reading of "Iceberg is" was, because both statements credit the format with something the writer decided. The first gap was a comparison of two defaults, ZSTD against Snappy, and the inverted gap was a comparison of two writers' encoding strategies. Neither is a fact about the table format's specification.
That left me unable to answer the storage question the way I'd framed it, which is itself the answer. Iceberg and DuckLake both store Parquet, and Parquet of a given logical content can land at a wide range of sizes depending on who wrote it and how, so the format name on the table tells you almost nothing about the bytes underneath. The only way to take the writer out of the storage comparison is the same move that fixes the read comparison: stop letting each format write its own files and register the same physical bytes into both catalogs. Then the Parquet on disk is identical and the only thing varying is the format's own metadata layer.
What the codec name hides
Compression is a strategy the writer runs.
The mental model worth replacing is the one where "compression" is a single setting you turn up or down like a quality slider, with the codec name standing in for the whole thing. But in Parquet the codec is the last and smallest step in a pipeline, and it operates on a byte stream the writer has already shaped through encoding. Dictionary encoding replaces values with small integer references into a table of distinct values, which is a large win on a low-cardinality column like a protocol name or an event type. On a high-cardinality one such as an ephemeral source port or a connection identifier, though, it's close to dead weight, because few values repeat, so you pay for the dictionary itself and get little dedup back. Run-length encoding collapses repeated runs, which depends entirely on whether the writer sorted or clustered the data so that runs exist at all. Page and column-chunk sizing changes how much context the compressor has to work with at once. Every one of those is a choice the writer makes, and only after they're made does ZSTD or Snappy get its turn on whatever's left.
So two writers that both honestly report "ZSTD level N" can hand that compressor very different inputs, and a smaller, more regular input compresses to fewer bytes regardless of the codec setting. On this data PyArrow was dictionary-encoding the high-cardinality columns that DuckDB chose to leave as plain values, so DuckDB's writer decided the dictionary wouldn't earn its keep, skipped it, and handed the compressor a smaller, more regular stream. That's why its file came out smaller at the same nominal level. None of that means PyArrow's writer is wrong, because dictionary-encoding aggressively is a reasonable default that pays off on many real schemas. PyArrow also exposes the controls to tune it if you write through its own API rather than through pyiceberg's wrapper. The narrower point is that the codec setting is a poor predictor of file size and the writer's encoding strategy is a strong one, so a size number means little until you know which writer produced it.
That is the same observation from the other end, because the encoding choices that move the bytes on disk also move query latency, since an engine reads what the writer laid down and an encoding that is cheap to scan is faster to filter. The storage divergence and the read divergence are not two findings, they are one finding seen from the write side and the read side. The writer is doing most of the work that benchmarks credit to the format or the engine, on size and on speed alike.
The only honest comparison
Write the files once, register the same bytes into both.
If the writer is the confound, then the way to measure the formats is to remove the writer from the comparison entirely. Both of these catalogs let you do exactly that, because both can adopt existing Parquet files rather than insisting on writing their own. So I wrote one canonical set of Parquet files a single time, then registered those same files into an Iceberg catalog with pyiceberg's add_files and into a DuckLake catalog with ducklake_add_data_files, and read both through DuckDB. The point of the design is that there is now exactly one set of bytes on disk and the two catalogs are pointing at it. Anything that differs between the two reads has to come from the format's metadata and read path rather than from the data, because the data is literally the same data.
At a billion rows, with the writer held out of the comparison this way, three of the four queries came back at parity. The filtered lookup landed at 1.00 times, the byte rollup at 1.01 times, and the subnet rollup at 1.01 times, all with a coefficient of variation at or under about 2.5 percent. A coefficient that low is well inside the run-to-run noise at that scale, and as close to "no difference" as a timing measurement gets to report. When the bytes are identical, the two table formats put the same data in front of DuckDB at the same speed. The 1.1-to-1.55-times gap from the first pass simply evaporated, because that gap was the codec default and the writer divergence the whole time and never the format.
I'll be straight about the one query that didn't come back at parity, because reporting only the three clean ones would be the dishonest version of this. A heavy GROUP BY with about 16.7 million distinct groups diverged by roughly 1.3 times between the two on the identical bytes. So the engines' read paths still differ on the hardest aggregation, the kind that builds an enormous hash table and is sensitive to how the scan feeds it. That's a real residual difference and I'm not going to wave it away, but it sits on the heaviest scan in the set rather than across the board. The shape of the result is that on identical bytes the formats read neutrally for the ordinary queries and diverge only at the extreme, which is a much narrower claim than "Iceberg is slower" and a much more accurate one.
Why the writer is the real variable
Read speed is set on the write path.
The reason this generalizes past my particular setup is that a columnar read is mostly a function of what's sitting on disk, and what's sitting on disk is decided when the file is written. The codec, the row-group size, the page size, whether a column is dictionary-encoded, how the statistics that drive predicate pushdown are laid out, all of that is fixed at write time and then read back over and over for the life of the file. The table format, by contrast, is the layer that tracks which files belong to the table and what their snapshots are. Once a query has resolved which files it needs, the format steps out of the way and the engine reads Parquet. So when you change the writer you change the thing the reader actually touches, and when you change only the format while keeping the same bytes, you've changed the bookkeeping and left the read alone. That's why the parity result is what I'd now expect rather than a surprise.
That reframes what you're choosing when you choose a table format, because you're not choosing read speed, and a benchmark that says you are has almost certainly let the two writers produce different bytes. The decision that actually moves read latency happens earlier, on the write path, in the encoder you use and the codec and row-group settings you give it. That decision is largely independent of whether the resulting files end up catalogued by Iceberg or by DuckLake. If you care about read speed, the place to spend your attention is the writer and its configuration. And the place to be skeptical is any chart that attributes a read difference to the format while using a different writer for each side without saying so.
None of which makes the format choice unimportant, it just moves the decision onto the axes where the formats genuinely differ. What you're really picking between Iceberg and DuckLake is the catalog and metadata model: how each handles the small-file problem and compaction, what its commit and concurrency story looks like, how its metadata scales as the table accumulates snapshots, and what tooling and engines can read it in your environment. Those are real and consequential differences, and they're where the decision belongs. So the practical move is to pick the format on the write path and the metadata properties you need and stop treating a read benchmark as the tiebreaker, because on identical bytes the read benchmark has very little to say.
What this is and isn't evidence of
One machine, one engine, ratios over absolutes
I want to bound this carefully, because it's the kind of result that's easy to over-read. The run is a single machine with one reader engine, DuckDB, against synthetic network-connection data, so the absolute times are a property of my hardware and mean nothing transplanted to yours. I'd treat the ratios as the transferable part rather than the milliseconds. The right way to read "three of four queries at parity, the fourth at about 1.3 times" is as a claim about relative behavior on identical bytes, not as a benchmark you could quote a latency figure from. I'd be the first to push back on anyone lifting the numbers out of that frame.
The storage gap carries the same caveat and one more, because the 193-against-114 divergence depends on the cardinality profile of these particular columns, since the whole mechanism turns on whether dictionary encoding helps or hurts on the data you actually have. On a corpus that's mostly low-cardinality the two writers would land much closer together, because dictionary encoding would earn its keep for both. So I'm not claiming PyArrow always writes larger files or that DuckDB is the better Parquet writer in general, only that on this data, at a matched codec, the writers differed by enough to dominate the format comparison I was trying to make. The durable part is the methodological one rather than the megabytes: the codec name is not the compression, matching it does not make a format comparison fair, and a size or latency number is not interpretable until you know which writer produced the file and how it was configured. All of that follows from how Parquet writers work rather than from my particular run.
There are also open questions I haven't closed. I read with DuckDB, and a different reader, Trino or DataFusion or ClickHouse, might lean on the format's metadata differently enough to break the parity I saw, so the neutrality result is specific to this reader until I run the others. The heavy high-cardinality GROUP BY that diverged deserves its own look, because I don't yet know whether that 1.3-times gap is a stable property of how each format feeds a large aggregation or an artifact of one run's scheduling. I'd rather say that plainly than fold it into the clean story. What I'm confident about is the narrower thing the design actually establishes, which is that the Iceberg-versus-DuckLake read gap I started with was a writer and codec artifact, and that on byte-identical files the formats read neutrally for ordinary queries on this setup.
The reason I trust that narrower claim is the design rather than the size of the effect. By writing the files once and registering the same bytes into both catalogs I removed the one variable that was big enough to swamp everything else. A parity result that survives that removal is more informative than a large difference that didn't control for it. The earlier 1.1-to-1.55-times gap was the louder number, but the quieter parity number is the one that's actually about the formats, and a quieter result you can defend beats a louder one you can't.
How not to get fooled by a read chart
Never trust a read benchmark that let the writers differ.
The practical takeaway I'd hand to anyone choosing a table format is that the read benchmark you've been shown is probably answering a different question than the one on its label. If the two sides were written by different encoders, or even by the same encoder at different codec defaults, then the chart is a comparison of writers wearing the formats' names. The only read benchmark worth believing is one that registers byte-identical files into both catalogs the way this run did. So the first question to ask of any format-versus-format read result is whether the bytes were the same. If the answer is no, or if the methodology doesn't say, the number tells you about the writers and not the formats, and you should treat it that way.
The same caution applies to a storage chart, because the size case fails in exactly the way the speed case does. When you read that format A is some percentage smaller than format B, the question that collapses most of the claim is which writer produced each file and at what settings. A ZSTD-versus-Snappy difference reported as an Iceberg-versus-DuckLake difference is the clean version of the trap. The subtler one is two files at the same codec and the same level that still differ because the writers encode differently, which doesn't even leave a settings difference for you to notice. If the methodology doesn't name the writer and its encoding configuration, the size number is describing a default, and a default is a choice somebody made for you rather than a fact about the format. The general version of how I keep a comparison like this from lying to me is in how to run a benchmark that doesn't lie. The short version is that you isolate the one variable you mean to study and make everything else identical, which here meant making the bytes identical before you let the clock run or weighed the file.
For security data the storage stakes are dollars, because storage is the recurring cost of keeping telemetry around long enough to investigate it. A 70 percent difference in on-disk size from the writer alone is the difference between a retention budget that holds a year and one that holds about seven months. The lever is real and yours to pull, because if your writer is bloating high-cardinality columns with dictionaries that don't earn their keep, the fix is in the writer's configuration and not in switching table formats. You'll get a better return tuning the encoding than migrating Iceberg to DuckLake or back. The format choice is a real decision for other reasons, but storage efficiency on a matched writer is mostly not one of them.
For security data the read stakes are ordinary engineering ones, but they're real too, because the format decision tends to get made early and lived with for years across a lot of telemetry. Making it on a read benchmark that was secretly measuring the encoder is how you end up ruling out a perfectly good option for a reason that was never true. Pick the format on the write path and the catalog and metadata behavior you actually need, tune read speed and storage where they're actually set, in the writer, and keep the questions apart so none of them gets answered with another one's evidence.
Evidence: Tier B (first-party, single machine; ratios transfer, absolute times don't). SDW Lab Iceberg-vs-DuckLake read comparison: one canonical Parquet set written once and registered into an Iceberg catalog via pyiceberg add_files and a DuckLake catalog via ducklake_add_data_files, read through DuckDB. At one billion rows three of four queries at parity (filtered 1.00×, byte_rollup 1.01×, subnet_rollup 1.01×, CV ≤ 2.5%); one 16.7M-distinct GROUP BY diverged ≈1.3×. The default-config gap (≈1.1–1.55× apparent Iceberg slowness) traced to Iceberg defaulting to ZSTD vs DuckLake's Snappy; at a matched codec PyArrow wrote ≈193 MB where DuckDB wrote ≈114 MB on identical data (≈70% larger, traced to PyArrow dictionary-encoding high-cardinality columns DuckDB left as plain values), with pyiceberg exposing no per-column encoding control to equalize it. At the larger scale the matched-codec DuckLake-written files were 1.14 GB against 1.93 GB for Iceberg, inverting the ZSTD-default-vs-Snappy-default size gap that had produced the "Iceberg is more storage-efficient" reading; the only writer-neutral comparison registers identical physical bytes into both catalogs. Methodology and the runnable comparison are published in the SDW Lab ocsf-read-scan benchmark; generality across other reader engines is an open follow-up.
Pick the format on the write path.
Over an open table format, the Parquet encoder sets both read speed and file size, because the format never touched either. When the bytes are identical, the formats read the same; and at a matched codec the writers still disagreed by almost a factor on size, which flipped which format looked smaller. The decision that's actually yours to make is the catalog and metadata model, and the benchmark worth believing, on speed or storage, is the one that made the bytes identical first.
