Writing · Pillar
Lakehouse foundations.
Open table formats and the interop layer beneath them. Iceberg, Delta, V3/V4 features, and the Arrow standards that make engine portability real.
Depth tier for ch5 (Performant) and ch2 (MOAR assembled) · 12 essays
Listed from the front door down to the measured evidence. Start here marks the entry essay. The reading-level tag says whether you are about to read the frame (Orientation), the argument (Depth), or the narrowest first-party finding (Evidence). The durability tag flags whether a number is durable architecture or an Evidence-pinned result tied to a version or price and meant to be re-run.
- Start here OrientationEvidence-pinned
Iceberg vs Delta Lake for security data.
Production evidence from Netflix (5 PB/day Iceberg), Insider (90% cost reduction), Adobe (5,000+ Delta tables), and InMobi (GDPR/CCPA on Delta) anchors a format decision that governs engine portability, vendor neutrality, and migration cost.
Read →
- DepthDurable
MOAR, explained.
What replaces and upgrades the ELK stack for security, component by component: Iceberg for the table, Arrow for the wire, D3FEND for the ontology, OCSF for the schema, Sigma for the detections, and a query engine you can change your mind about. Modular Open Architecture explained, and why being able to swap any one piece without rebuilding the rest is what makes it safe to adopt.
Read →
- DepthEvidence-pinned
Iceberg V3 changed the security lakehouse thesis.
Earlier essays argued Iceberg over Delta on vendor-neutrality. V3's Puffin deletion vectors, Variant type, row-level lineage, and default values plus V4 proposals change the trade-off space. Some prior recommendations need revision; others are reinforced.
Read →
- DepthEvidence-pinned
V4 relative paths vs DuckLake's database-metadata.
Different facets of the same problem space, not the same fight. What V4's relative paths solve, where DuckLake's database-as-metadata wins, and which security-data workloads each fits.
Read →
- DepthEvidence-pinned
Iceberg table maintenance at scale.
Compaction, snapshot expiration, orphan file cleanup. What goes wrong at petabyte scale, what the maintenance budget looks like, and where vendor catalogs help or hurt.
Read →
- DepthEvidence-pinned
Deletion vectors and GDPR.
Iceberg's right-to-erasure story, honestly. What Puffin deletion vectors actually deliver, where the gaps remain, and what an audit looks like.
Read →
- DepthEvidence-pinned
Variant type ends the flattening wars.
The Iceberg V3 Variant type is the structural fix for what flattening anti-patterns broke. CloudTrail nested JSON, EDR alerts, anything semi-structured: store it native, query it native.
Read →
- DepthEvidence-pinned
Row lineage as the missing CDC primitive.
Iceberg V3 row lineage may close the audit-trail gap that has dogged lakehouse detection engineering. What works Spark-side today, what's blocked on pyiceberg + Nessie, what an end-to-end CDC pattern looks like.
Read →
- DepthEvidence-pinned
Arrow and ADBC: a foundational pillar.
Why Arrow Database Connectivity is the wire-protocol layer that lets you swap engines without rewriting client code. The columnar-throughput case versus JDBC/ODBC.
Read →
- DepthEvidence-pinned
Arrow Flight and Flight SQL.
The columnar wire protocol for security data. Where Flight SQL fits in a federated stack, what it costs at the analyst's tool layer, and which engines actually speak it.
Read →
- EvidenceEvidence-pinned
The encoder is the read lever, not the table format.
Register byte-identical Parquet into both an Iceberg and a DuckLake catalog, read both with DuckDB, and at a billion rows three of four queries come back at parity. The read-speed difference people attribute to the format is the Parquet writer: at a matched codec PyArrow writes 193 MB where DuckDB writes 114 MB on the same data. Pick the format on the write path, not a read benchmark.
Read →
- EvidenceEvidence-pinned
The write pattern is the architectural decision.
Iceberg and DuckLake are nearly read-neutral on identical data but diverge sharply on how data enters. Over a 10-to-200-file ladder Iceberg's planning grows 17.6x while DuckLake's SQL-catalog stays flat; on tiny streaming commits DuckLake's inlining runs 2-4x faster with zero data files. Map the format to the write contract: streaming hot tier versus bulk and forensic cold tier.
Read →