Writing · Lakehouse internals
The encoder is the read lever, not the table format.
I went into this expecting to find a read-speed difference between Iceberg and DuckLake, because that's the comparison everyone reaches for when they pick a table format, and I'd seen Iceberg come back slower in my own earlier runs. What I found instead is that the table format was never the thing I was measuring. The lever on read speed is the Parquet writer that produced the files, and once I took the writer out of the comparison by registering the byte-identical bytes into both catalogs, the two formats read the same. The same thing turned out to be true of file size, where two writers handed the identical rows and the identical codec still disagreed by almost a factor, so the encoder governs both the read and the bytes on disk while the format is the catalog around them.
What I thought I was measuring
Is Iceberg slower to read than DuckLake?
The two table formats sit at different points on the spectrum, and the way you usually decide between them is by reputation: Iceberg is the heavier specification with the broad catalog ecosystem and the manifest-based metadata layer, while DuckLake keeps its table metadata in a SQL database and aims to be the lighter, lower-ceremony option. When someone asks me which one reads security telemetry faster, the honest starting position is that I didn't know, so I set up a scale ladder to find out, generating synthetic network-connection events and querying them through DuckDB against an Iceberg table and against a DuckLake table at one million, ten million, a hundred million, and a billion rows.
On the first pass Iceberg looked slower, and not by a trivial amount. Across the query set it came back somewhere between about 1.1 times and 1.55 times slower than DuckLake on the same logical data, which is the kind of gap that would feel decisive if you stopped there and wrote it up. If I'd published that table it would have read as a clean format-versus-format result, with DuckLake winning the read path, and a practitioner deciding between the two on read latency would have taken Iceberg off the shortlist on the strength of it. The number was real in the sense that the clock genuinely measured it. It just wasn't measuring what the headline would have said it was measuring.
What stopped me from writing it up that way is that the two setups weren't reading the same files. Each format had written its own Parquet through its own path, so when I compared read latencies I was comparing two different sets of bytes that happened to encode the same rows, and any difference between them could just as easily live in the bytes as in the format's read machinery. Before I could say anything about the formats I had to make the files identical, which turned out to be the whole story.
Where the gap actually lived
Iceberg defaulted to ZSTD, DuckLake to Snappy.
The first confound was the easy one to name once I looked. The two paths weren't even compressing the data the same way, because pyiceberg's write path defaulted the files to ZSTD while the DuckLake path wrote Snappy, and ZSTD trades decompression work for smaller files in a way that shows up directly on a scan-heavy read. So a chunk of the apparent Iceberg slowness was a codec default sitting underneath the format, not a property of the format itself, and the moment you say "I measured Iceberg against DuckLake" while one is on ZSTD and the other is on Snappy, you've already mislabeled the result. The format names were on the axis, but the codec was doing the work.
The fix that seems obvious is to set both writers to the same codec and call it controlled, and that's where the second and more stubborn confound surfaced. Matching the codec does not match the bytes, because the two Parquet writers disagree about everything below the codec. On the identical input data, with the codec held equal, PyArrow's writer produced a file of about 193 MB where DuckDB's writer produced about 114 MB for the same rows, which is a roughly 1.7-times difference in size that has nothing to do with compression algorithm and everything to do with encoding decisions. The largest single contributor I could see is dictionary encoding on the high-cardinality columns, where PyArrow makes different choices about when to dictionary-encode and when to fall back, and pyiceberg gives you no per-column control to override it, so you can't simply tell it to encode the way DuckDB does.
That second confound is the one I'd want a benchmark reader to internalize, because "same codec" feels like it should mean "same bytes" and it doesn't. Two correct Parquet writers, handed the identical rows and the identical compression codec, will still emit substantially different files, and a file that's 1.7 times larger has more to read off disk, more to decompress, and a different layout for the scanner to walk. If that difference is allowed to ride along inside a comparison labeled Iceberg-versus-DuckLake, then the comparison is measuring the writers and reporting on the formats, which is exactly the kind of mislabeled result that makes published read benchmarks untrustworthy.
The same writer governs the bytes on disk
"Iceberg is more storage-efficient" was a codec default that flipped.
I'd actually walked into the same trap on storage before I got to read speed, because the first version of this work set out to answer a different question, which was simply whether Iceberg or DuckLake stored the synthetic network-connection corpus more cheaply, and storage is a real cost line when you keep security telemetry for a year or more and the per-partition difference compounds across the whole retention window. The early numbers told the story everyone tells, that the Iceberg table came out meaningfully smaller, which lined up with the conventional read that Iceberg is the more storage-efficient format that the mature ecosystem has tuned hardest. The reason I didn't write that down is the same reason the read result fell apart: pyiceberg defaulted its files to ZSTD while the DuckLake path defaulted to Snappy, so I was comparing a ZSTD file against a Snappy file and calling the difference a property of the format, which is a codec comparison wearing a format comparison's clothes.
When I held the codec constant and let each format use its own default writer, the ranking inverted. At the larger scale I re-ran to confirm it, the DuckLake-written files came out the smaller of the two, 1.14 GB against 1.93 GB for the same dataset, so the format people call more storage-efficient was, on a matched codec, writing the larger files, because PyArrow's writer made encoding choices that left more bytes on disk than DuckDB's did. The honest reading of that is not "DuckLake is the more efficient format" any more than the original reading of "Iceberg is" was honest, because both statements credit the format with something the writer decided. The first gap was a comparison of two defaults, ZSTD against Snappy, and the inverted gap was a comparison of two writers' encoding strategies, and neither is a fact about the table format's specification.
That left me unable to answer the storage question the way I'd framed it, which is itself the answer. Iceberg and DuckLake both store Parquet, and Parquet of a given logical content can land at a wide range of sizes depending on who wrote it and how, so the format name on the table tells you almost nothing about the bytes underneath, and the only way to take the writer out of the storage comparison is the same move that fixes the read comparison: stop letting each format write its own files and register the same physical bytes into both catalogs, so that the Parquet on disk is identical and the only thing varying is the format's own metadata layer.
What the codec name hides
Compression is a strategy the writer runs, not a single knob.
The mental model worth replacing is the one where "compression" is a single setting you turn up or down like a quality slider, with the codec name standing in for the whole thing, because in Parquet the codec is the last and smallest step in a pipeline and it operates on a byte stream the writer has already shaped through encoding. Dictionary encoding replaces values with small integer references into a table of distinct values, which is a large win on a low-cardinality column like a protocol name or an event type and close to dead weight on a high-cardinality one such as an ephemeral source port or a connection identifier, where you pay for the dictionary itself and get little dedup back because few values repeat. Run-length encoding collapses repeated runs, which depends entirely on whether the writer sorted or clustered the data so that runs exist at all, and page and column-chunk sizing changes how much context the compressor has to work with at once. Every one of those is a choice the writer makes, and only after they're made does ZSTD or Snappy get its turn on whatever's left.
So two writers that both honestly report "ZSTD level N" can hand that compressor very different inputs, and a smaller, more regular input compresses to fewer bytes regardless of the codec setting. On this data PyArrow was dictionary-encoding the high-cardinality columns that DuckDB chose to leave as plain values, so DuckDB's writer decided the dictionary wouldn't earn its keep, skipped it, and handed the compressor a smaller, more regular stream, which is why its file came out smaller at the same nominal level. None of that means PyArrow's writer is wrong, because dictionary-encoding aggressively is a reasonable default that pays off on many real schemas, and PyArrow exposes the controls to tune it if you write through its own API rather than through pyiceberg's wrapper. The narrower point is that the codec setting is a poor predictor of file size and the writer's encoding strategy is a strong one, so a size number means little until you know which writer produced it.
That is the same observation from the other end, because the encoding choices that move the bytes on disk also move query latency, since an engine reads what the writer laid down and an encoding that is cheap to scan is faster to filter. The storage divergence and the read divergence are not two findings, they are one finding seen from the write side and the read side: the writer is doing most of the work that benchmarks credit to the format or the engine, on size and on speed alike.
The only honest comparison
Write the files once, register the same bytes into both.
If the writer is the confound, then the way to measure the formats is to remove the writer from the comparison entirely, and both of these catalogs let you do exactly that because both can adopt existing Parquet files rather than insisting on writing their own. So I wrote one canonical set of Parquet files a single time, then registered those same files into an Iceberg catalog with pyiceberg's add_files and into a DuckLake catalog with ducklake_add_data_files, and read both through DuckDB. The point of the design is that there is now exactly one set of bytes on disk and the two catalogs are pointing at it, so anything that differs between the two reads has to come from the format's metadata and read path rather than from the data, because the data is literally the same data.
At a billion rows, with the writer held out of the comparison this way, three of the four queries came back at parity. The filtered lookup landed at 1.00 times, the byte rollup at 1.01 times, and the subnet rollup at 1.01 times, all with a coefficient of variation at or under about 2.5 percent, which is well inside the run-to-run noise at that scale and is as close to "no difference" as a timing measurement gets to report. When the bytes are identical, the two table formats put the same data in front of DuckDB at the same speed, and the 1.1-to-1.55-times gap from the first pass simply evaporated, because that gap was the codec default and the writer divergence the whole time and never the format.
I'll be straight about the one query that didn't come back at parity, because reporting only the three clean ones would be the dishonest version of this. A heavy GROUP BY with about 16.7 million distinct groups diverged by roughly 1.3 times between the two on the identical bytes, so the engines' read paths still differ on the hardest aggregation, the kind that builds an enormous hash table and is sensitive to how the scan feeds it. That's a real residual difference and I'm not going to wave it away, but it sits on the heaviest scan in the set rather than across the board, and the shape of the result is that on identical bytes the formats read neutrally for the ordinary queries and diverge only at the extreme, which is a much narrower claim than "Iceberg is slower" and a much more accurate one.
Why the writer is the real variable
Read speed is set on the write path.
The reason this generalizes past my particular setup is that a columnar read is mostly a function of what's sitting on disk, and what's sitting on disk is decided when the file is written. The codec, the row-group size, the page size, whether a column is dictionary-encoded, how the statistics that drive predicate pushdown are laid out, all of that is fixed at write time and then read back over and over for the life of the file. The table format, by contrast, is the layer that tracks which files belong to the table and what their snapshots are, and once a query has resolved which files it needs, the format steps out of the way and the engine reads Parquet. So when you change the writer you change the thing the reader actually touches, and when you change only the format while keeping the same bytes, you've changed the bookkeeping and left the read alone, which is why the parity result is what I'd now expect rather than a surprise.
That reframes what you're choosing when you choose a table format, because you're not choosing read speed, and a benchmark that says you are has almost certainly let the two writers produce different bytes. The decision that actually moves read latency happens earlier, on the write path, in the encoder you use and the codec and row-group settings you give it, and that decision is largely independent of whether the resulting files end up catalogued by Iceberg or by DuckLake. If you care about read speed, the place to spend your attention is the writer and its configuration, and the place to be skeptical is any chart that attributes a read difference to the format while quietly using a different writer for each side.
None of which makes the format choice unimportant, it just moves the decision onto the axes where the formats genuinely differ. What you're really picking between Iceberg and DuckLake is the catalog and metadata model: how each handles the small-file problem and compaction, what its commit and concurrency story looks like, how its metadata scales as the table accumulates snapshots, and what tooling and engines can read it in your environment. Those are real and consequential differences, and they're where the decision belongs, so the practical move is to pick the format on the write path and the metadata properties you need and stop treating a read benchmark as the tiebreaker, because on identical bytes the read benchmark has very little to say.
What this is and isn't evidence of
One machine, one engine, ratios over absolutes.
I want to bound this carefully, because it's the kind of result that's easy to over-read. The run is a single machine with one reader engine, DuckDB, against synthetic network-connection data, so the absolute times are a property of my hardware and mean nothing transplanted to yours, and I'd treat the ratios as the transferable part rather than the milliseconds. The right way to read "three of four queries at parity, the fourth at about 1.3 times" is as a claim about relative behavior on identical bytes, not as a benchmark you could quote a latency figure from, and I'd be the first to push back on anyone lifting the numbers out of that frame.
The storage gap carries the same caveat and one more, because the 193-against-114 divergence depends on the cardinality profile of these particular columns, since the whole mechanism turns on whether dictionary encoding helps or hurts on the data you actually have. On a corpus that's mostly low-cardinality the two writers would land much closer together, because dictionary encoding would earn its keep for both, so I'm not claiming PyArrow always writes larger files or that DuckDB is the better Parquet writer in general, only that on this data, at a matched codec, the writers differed by enough to dominate the format comparison I was trying to make. The durable part is the methodological one rather than the megabytes: the codec name is not the compression, matching it does not make a format comparison fair, and a size or latency number is not interpretable until you know which writer produced the file and how it was configured, which follows from how Parquet writers work rather than from my particular run.
There are also honest open questions I haven't closed. I read with DuckDB, and a different reader, Trino or DataFusion or ClickHouse, might lean on the format's metadata differently enough to break the parity I saw, so the neutrality result is specific to this reader until I run the others. The heavy high-cardinality GROUP BY that diverged deserves its own look, because I don't yet know whether that 1.3-times gap is a stable property of how each format feeds a large aggregation or an artifact of one run's scheduling, and I'd rather say that plainly than fold it into the clean story. What I'm confident about is the narrower thing the design actually establishes, which is that the Iceberg-versus-DuckLake read gap I started with was a writer and codec artifact, and that on byte-identical files the formats read neutrally for ordinary queries on this setup.
The reason I trust that narrower claim is the design rather than the size of the effect. By writing the files once and registering the same bytes into both catalogs I removed the one variable that was big enough to swamp everything else, and a parity result that survives that removal is more informative than a large difference that didn't control for it. The earlier 1.1-to-1.55-times gap was the louder number, but the quieter parity number is the one that's actually about the formats, and a quieter result you can defend beats a louder one you can't.
How not to get fooled by a read chart
Never trust a read benchmark that let the writers differ.
The practical takeaway I'd hand to anyone choosing a table format is that the read benchmark you've been shown is probably answering a different question than the one on its label. If the two sides were written by different encoders, or even by the same encoder at different codec defaults, then the chart is a comparison of writers wearing the formats' names, and the only read benchmark worth believing is one that registers byte-identical files into both catalogs the way this run did. So the first question to ask of any format-versus-format read result is whether the bytes were the same, and if the answer is no, or if the methodology doesn't say, the number tells you about the writers and not the formats and you should treat it that way.
The same caution applies to a storage chart, because the size case fails in exactly the way the speed case does. When you read that format A is some percentage smaller than format B, the question that collapses most of the claim is which writer produced each file and at what settings, because a ZSTD-versus-Snappy difference reported as an Iceberg-versus-DuckLake difference is the clean version of the trap and two files at the same codec and the same level that still differ because the writers encode differently is the subtler one that doesn't even leave a settings difference for you to notice. If the methodology doesn't name the writer and its encoding configuration, the size number is describing a default, and a default is a choice somebody made for you rather than a fact about the format. The general version of how I keep a comparison like this from lying to me is in how to run a benchmark that doesn't lie, and the short version is that you isolate the one variable you mean to study and make everything else identical, which here meant making the bytes identical before you let the clock run or weighed the file.
For security data the storage stakes are dollars, because storage is the recurring cost of keeping telemetry around long enough to investigate it, and a 70 percent difference in on-disk size from the writer alone is the difference between a retention budget that holds a year and one that holds about seven months. The lever is real and yours to pull, because if your writer is bloating high-cardinality columns with dictionaries that don't earn their keep, the fix is in the writer's configuration and not in switching table formats, and you'll get a better return tuning the encoding than migrating Iceberg to DuckLake or back. The format choice is a real decision for other reasons, but storage efficiency on a matched writer is mostly not one of them.
For security data the read stakes are ordinary engineering ones rather than anything exotic, but they're real too, because the format decision tends to get made early and lived with for years across a lot of telemetry, and making it on a read benchmark that was secretly measuring the encoder is how you end up ruling out a perfectly good option for a reason that was never true. Pick the format on the write path and the catalog and metadata behavior you actually need, tune read speed and storage where they're actually set, in the writer, and keep the questions apart so none of them gets answered with another one's evidence.
Evidence: Tier B (first-party, single machine; ratios transfer, absolute times don't). SDW Lab Iceberg-vs-DuckLake read comparison: one canonical Parquet set written once and registered into an Iceberg catalog via pyiceberg add_files and a DuckLake catalog via ducklake_add_data_files, read through DuckDB. At one billion rows three of four queries at parity (filtered 1.00×, byte_rollup 1.01×, subnet_rollup 1.01×, CV ≤ 2.5%); one 16.7M-distinct GROUP BY diverged ≈1.3×. The default-config gap (≈1.1–1.55× apparent Iceberg slowness) traced to Iceberg defaulting to ZSTD vs DuckLake's Snappy; at a matched codec PyArrow wrote ≈193 MB where DuckDB wrote ≈114 MB on identical data (≈70% larger, traced to PyArrow dictionary-encoding high-cardinality columns DuckDB left as plain values), with pyiceberg exposing no per-column encoding control to equalize it. At the larger scale the matched-codec DuckLake-written files were 1.14 GB against 1.93 GB for Iceberg, inverting the ZSTD-default-vs-Snappy-default size gap that had produced the "Iceberg is more storage-efficient" reading; the only writer-neutral comparison registers identical physical bytes into both catalogs. Methodology and the runnable comparison are published in the SDW Lab ocsf-read-scan benchmark; generality across other reader engines is an open follow-up.
Pick the format on the write path, not the read or storage chart.
Over an open table format, the Parquet encoder sets both read speed and file size, not Iceberg-versus-DuckLake. When the bytes are identical, the formats read the same; and at a matched codec the writers still disagreed by almost a factor on size, which flipped which format looked smaller. The decision that's actually yours to make is the catalog and metadata model, and the benchmark worth believing, on speed or storage, is the one that made the bytes identical first.