Economics deep-dive

The index pays twice.

A hot search index stores more bytes per event than a columnar lakehouse table, and it stores those bytes on more expensive storage, so it loses on both factors at once and they multiply. On the same ten million Zeek connection events, an OpenSearch index on block storage costs about 14.8× a warm Iceberg-on-S3 table, and the gap is mostly the storage class rather than the codec. At thirty days that is a rounding error on a SOC budget; at the seven-year retention horizon a regulated firm actually has to plan for, it is the difference between a line item and a project.

Reading time: about 16 minutes. Evidence tier: B for the measured footprints, C for the desk-priced retention curves, and I label which is which as I go. The footprints are first-party measurements on a single host over the same sha256-pinned 10M-row Zeek conn corpus as the flagship rerun; the prices are AWS us-east-1 list, read off the live rate card on 2026-06-10 and re-checked unchanged on 2026-06-14. This is a storage floor, not a total-cost-of-ownership model, and I keep those two layers separate throughout because conflating them is how cost arguments go wrong. The byte ratios are also parameters of one corpus, so I re-measured two more to show how far they move, and the short version is that the shape of the finding travels while the exact multiplier does not.

The instinct, when the storage bill is too high, is to squeeze the bytes harder, and the measurement says that instinct is aiming at the smaller lever. Tuned ClickHouse with a blanket ZSTD(22) codec actually compresses better than Iceberg's zstd default, 9.0× against 8.5×, so on bytes alone it should win. It still costs 3.3× more per month, because those well-compressed bytes are sitting on block storage to keep the hot OLAP table fast, while Iceberg's slightly larger bytes are sitting on S3. The better codec lost to the storage class.

That is the finding I would carry out of this whole exercise: compression tuning moves the bill by tens of percent, and moving bytes from gp3 to S3 moves it by 3.5×, and moving them to Glacier Instant Retrieval moves it by twenty. The architecture decision, which is where old data is allowed to live, dominates the engineering decision, which is how hard you squeeze it. A team can spend a quarter tuning codecs and dictionary encodings and shave a fifth off the bill, or it can decide that data older than a few weeks belongs on object storage and shave most of it off, and the second decision is usually less work than the first.

It also reframes what an open table format buys you. The case for Iceberg or Delta is often pitched as portability, which is real, but the cost-to-serve angle is more immediate: an open table on object storage is the thing that lets old data fall to a cheaper storage class without losing queryability, and that fall is where the money is. A vendor-managed hot store that you can only read through the vendor's compute gives you that lever only on the vendor's terms, because even when it does tier the bytes to cheaper storage you can still read them back through nothing but the compute the vendor sells you, so the cold data never leaves the vendor's price list even after it leaves the expensive disk.

A multiplier on its own does not tell you whether to care. What makes the 14.8× matter is that both layers are linear in days, so the gap grows with the retention window, and the retention window for a regulated firm is not thirty days. Priced at one terabyte per day of raw ingest, held at steady state, storage only, the curves look like this:

At thirty days the index-versus-lakehouse gap is about $1,100 a month, the difference between $1,200 on the index and $81 on warm Iceberg. That is the kind of number a SOC absorbs without a meeting. At the seven-year horizon a regulated firm under rules like the SEC's 17a-4 and the EU's DORA has to plan for, the same gap is about $95,000 a month per terabyte per day of ingest, $102,200 against $6,914, and that is the kind of number that starts a migration. Nothing about the architecture changed between those two columns; only the retention window did, and the window is set by a regulator rather than by an engineer.

One more thing about the units, because they decide how alarming the table is for any given reader. Everything here is normalized to one terabyte per day of raw ingest, which is a mid-size enterprise SOC, and a large financial-services firm often runs several times that, so a shop ingesting five terabytes a day multiplies every figure in the table by five and the seven-year index-versus-lakehouse gap goes from $95,000 a month to most of half a million. The linearity that makes the tiering arithmetic clean makes the volume arithmetic clean too, so you can read your own number straight off the table by scaling it to your ingest.

There is a way to read that $102,200 as a choice rather than a fact, and it is the tiered design the table is really arguing for. Because both layers are linear in days, a real architecture does not pay the all-hot rate across the whole window; it keeps the recent thirty days on the index, where the $1,200 buys the instant lookups an analyst actually uses, and lets the remaining years fall to a cold class. The thirty-day index window is $1,200 a month and the full seven years of cold Parquet on Glacier Instant Retrieval is $1,055, so the tiered bill lands a little over $2,000 a month against $102,200 for keeping everything hot, roughly forty-five times less for the same retained data. That spread is the budget the architecture hands back, and it is the honest comparison hiding inside the table, because the all-hot column is not a design anyone defends out loud but what a platform costs once tiering gets hard enough that teams stop doing it.

The cold tier widens it further. Index-on-gp3 against cold Parquet on Glacier Instant Retrieval is 96.7× on storage alone, which is the largest number in the table and the most easily abused, so it comes with a condition: Glacier IR charges $0.03 per GB on retrieval, so that tier is for data you genuinely rarely read, and any claim that cites the 96.7× has to either stay storage-only or model how often the data actually gets pulled back. The honest version of the cold-tier number is that it is enormous for archival you touch a few times a year and shrinks fast if your analysts are querying it weekly.

I want to be precise about what this measures, because storage-cost arguments earn their bad reputation by sliding from one thing into everything. This is the storage floor and only the storage floor: no compute, no licensing, no ops labor, no egress, no IOPS or throughput add-ons beyond the gp3 baseline. A real platform spends on all of those, and on some workloads the compute and license lines dwarf the storage line, so the 14.8× is a floor under the storage share of the bill rather than a claim about the whole bill. The reason the storage share is still worth isolating is that it is the share retention compounds, so it is the part of the bill that grows on its own while everything else holds roughly flat per query.

The byte ratios are also parameters of this particular corpus, as the previous section measured rather than asserted, and they will move on other data. Nested OCSF with deep object structure, or long free-text message fields, compress differently in an index than in columnar Parquet, sometimes narrowing the footprint gap and sometimes widening it, so the right move in an engagement is to re-measure on the actual workload rather than to carry these exact ratios across. The shape of the finding, that the storage class dominates the codec and that retention is the multiplier, is the portable part; the specific 4.2× is not.

The curves also assume a steady state, a flat terabyte a day held at constant volume, and real ingest rarely holds flat. A growing program adds feeds and endpoints year over year, so its retention curve bends steeper than the linear one priced here and the all-hot bill compounds faster than the table suggests rather than slower. That steady state is the conservative direction for the lakehouse argument, because anything that makes the window longer or the daily volume larger widens the gap the table already shows, so I priced the flat case and left the reader to scale it up.

And no real architecture should run everything on one tier. Hardly anyone would choose to serve seven years of data from a hot index on purpose, so the all-hot column is a strawman if you read it as a deliberate proposal, and the reason it still shows up in the wild is that teams get backed into it. The honest comparison the table enables is a tiered design, a hot tier for the recent seven-to-thirty days plus warm and cold for the long tail, against an all-hot design that keeps everything on the index because the platform makes tiering hard. That second design is more common than it should be, because a lot of SIEM pricing couples ingestion and retention so that old data keeps paying the hot rate, and the table is really a measurement of what that coupling costs.

Finally, this is not the old desk-derived 130–227× storage-cost ratio I have used before, and these numbers do not confirm or replace it. That number came from list-price arithmetic, Splunk's indexed-storage rate of roughly $3 to $10 per GB-month against S3 Standard's $0.023, which pencils out anywhere from about 130× to 435× and which I have carried conservatively at 130× to 227×, so it was always storage list price over storage list price, with compute and license sitting on top of it rather than inside it. This benchmark measures bytes and storage prices directly, and what it does is put a measured storage floor of roughly 15× to 97×, depending on tier, under that same storage ratio, which is a smaller and more defensible thing to stand on than the desk arithmetic.