MOAR Stack · Performant
Table format & wire (Iceberg + Arrow).
The open table any engine can read off cheap object storage, and the columnar wire that moves data between tools without a parse tax at each hop. Lab tests at a billion rows found that the Parquet writer governs both read speed and file size, not the table format: byte-identical files read at parity across most queries, and at a matched codec PyArrow wrote 193 MB to DuckDB's 114 MB on the same data, so the usual claim that Iceberg is more storage-efficient than DuckLake turns out to be measuring the writer.
Performant component · 11 essays
Listed from the front door down to the measured evidence. Start here marks the entry essay. The reading-level tag says whether you are reading the frame (Orientation), the argument (Depth), or a first-party finding (Evidence). The durability tag flags durable architecture versus an Evidence-pinned result tied to a version or price.
- Start here OrientationEvidence-pinned
Iceberg vs Delta Lake for security data.
Production evidence from Netflix (5 PB/day Iceberg), Insider (90% cost reduction), Adobe (5,000+ Delta tables), and InMobi (GDPR/CCPA on Delta) anchors a format decision that governs engine portability, vendor neutrality, and migration cost.
Read →
- DepthEvidence-pinned
Iceberg V3 changed the security lakehouse thesis.
Earlier essays argued Iceberg over Delta on vendor-neutrality. V3's Puffin deletion vectors, Variant type, row-level lineage, and default values plus V4 proposals change the trade-off space. Some prior recommendations need revision; others are reinforced.
Read →
- DepthEvidence-pinned
V4 relative paths vs DuckLake's database-metadata.
Different facets of the same problem space, not the same fight. What V4's relative paths solve, where DuckLake's database-as-metadata wins, and which security-data workloads each fits.
Read →
- DepthEvidence-pinned
Iceberg table maintenance at scale.
Compaction, snapshot expiration, orphan file cleanup. What goes wrong at petabyte scale, what the maintenance budget looks like, and where vendor catalogs help or hurt.
Read →
- DepthEvidence-pinned
Deletion vectors and GDPR.
Iceberg's right-to-erasure story, honestly. What Puffin deletion vectors actually deliver, where the gaps remain, and what an audit looks like.
Read →
- DepthEvidence-pinned
Variant type ends the flattening wars.
The Iceberg V3 Variant type is the structural fix for what flattening anti-patterns broke. CloudTrail nested JSON, EDR alerts, anything semi-structured: store it native, query it native.
Read →
- DepthEvidence-pinned
Row lineage as the missing CDC primitive.
Iceberg V3 row lineage may close the audit-trail gap that has dogged lakehouse detection engineering. What works Spark-side today, what's blocked on pyiceberg + Nessie, what an end-to-end CDC pattern looks like.
Read →
- DepthEvidence-pinned
Arrow and ADBC: a foundational pillar.
Why Arrow Database Connectivity is the wire-protocol layer that lets you swap engines without rewriting client code. The columnar-throughput case versus JDBC/ODBC.
Read →
- DepthEvidence-pinned
Arrow Flight and Flight SQL.
The columnar wire protocol for security data. Where Flight SQL fits in a federated stack, what it costs at the analyst's tool layer, and which engines actually speak it.
Read →
- EvidenceEvidence-pinned
The encoder is the read lever, not the table format.
Register byte-identical Parquet into both an Iceberg and a DuckLake catalog, read both with DuckDB, and at a billion rows three of four queries come back at parity. The read-speed difference people attribute to the format is the Parquet writer: at a matched codec PyArrow writes 193 MB where DuckDB writes 114 MB on the same data. Pick the format on the write path, not a read benchmark.
Read →
- EvidenceEvidence-pinned
The write pattern is the architectural decision.
Iceberg and DuckLake are nearly read-neutral on identical data but diverge sharply on how data enters. Over a 10-to-200-file ladder Iceberg's planning grows 17.6x while DuckLake's SQL-catalog stays flat; on tiny streaming commits DuckLake's inlining runs 2-4x faster with zero data files. Map the format to the write contract: streaming hot tier versus bulk and forensic cold tier.
Read →
Capability Matrix
The Matrix scores format openness, engine reach, and the write-path mechanics.
See the Capability Matrix →