Economics deep-dive

The index in your bucket.

A colleague in San Francisco told me there was a category of security data infrastructure I should be watching, that it was called "scanners," that scanner.dev had led it, and that the same architecture had been showing up in enough places to have a real following. I take tips like that seriously, because a category name is a compressed claim about architecture, and if the industry had quietly agreed on a new one for security data I wanted to know what the agreement said. So I went looking for the category, and what I found was more interesting than what I was told I'd find: the architecture is real, it is older than the company said to have led it, the following is real too though it belongs to a different set of players, and the name does not exist at all.

Reading time: about 12 minutes. Evidence tiers are labeled inline as they appear: Tier A for the peer literature, Tier B for my lab's first-party measurements, Tier C for vendor-authored claims, flagged as such. Two arguments here, the lock-in-moved argument and the cross-engine correctness question, are my own inferences rather than reported findings, and the text marks them where they start.

That move is worth stating plainly, because it really does recur. Keep the telemetry in object storage, usually the customer's own bucket, and bring compute to it only when a question arrives, instead of ingesting everything into a vendor's hot store and paying a per-gigabyte toll at the door. Scanner's version is a clean example of the mechanics: it reads raw logs from your S3 buckets, builds a compact index (posting lists for text, range structures for numbers, roughly 150 GB of index per terabyte of uncompressed logs, per its own docs), writes those index files back into a bucket in your AWS account, and answers queries by fanning out short-lived Lambda workers that scan only the byte-ranges the index says matter. The company claims scan rates in the hundreds of gigabytes per second (Tier C, vendor docs). It's good engineering, and the pricing that comes with it undercuts ingest-licensed platforms by a wide margin on the workloads it fits.

But Scanner joined this architecture rather than leading it. ChaosSearch was building a proprietary index over customer S3 in 2017 and went GA in 2019. Hydrolix started in 2018 on decoupled object storage with stateless Kubernetes compute. Humio was serving index-free log search around 2016, early enough that CrowdStrike bought it in 2021. Quickwit, an Apache-licensed Rust engine doing decoupled search over object storage, launched in 2020 and was acquired by Datadog in January 2025. Cribl Search went GA in November 2022 with the search-in-place framing and no persistent index at all. Scanner was founded in 2022, into a field where every one of those already existed.

And the popular following my colleague described is real, though it attaches to the pattern rather than to that company. The community moment I can actually measure belongs to Matano, the open-source, Iceberg-native security lake out of YC's winter 2023 batch, whose Launch HN drew 140 points and 53 comments in January 2023. Scanner.dev has never cleared six points on Hacker News with any post; its own Series A announcement drew four points and zero comments. What Scanner does have, and it matters, is capital conviction: a $22M Series A led by Sequoia in March 2026, with CRV, Mantis, and angels whose names carry weight in exactly this space, Christina Cacioppo of Vanta, Tom Killalea, who was Amazon's first CISO, and Venkat Venkataramani of Rockset. When I trace the "having a moment" feeling to its sources, that's what I find: the money arrived loudly in 2026, and the community arrived quietly years earlier, somewhere else.

I've been describing the security data platform market as a choice between two builds: the index-as-store design, where your data and your query engine are one purchase and one decision, and the open-table design, where the data sits in a format any engine can read and the engine is a replaceable part. The research for this essay convinced me there's a third build sitting between them, and that the vocabulary confusion my colleague ran into exists partly because few have named it as its own thing.

In the middle build, the raw logs stay in your bucket, in your account, at object-storage prices, and everything the vendors say about that is true. But the derived artifact, the index that makes those logs searchable at interactive speed, is a proprietary format that only the vendor's compute can interpret. Scanner's docs describe the index functionally and publish no format specification; I found no indication that any third-party engine can read a Scanner index file, and no roadmap language about Iceberg, Delta, or any open table format. ChaosSearch patented its index, and like Scanner's it sits beside raw logs you still hold. Hydrolix and Axiom sit closer to the first build, since they transform on ingest into proprietary columnar formats (HDX, Axiom's store) that become the system of record rather than a sidecar, so at exit you own their files, not your raw telemetry. (Cribl Search is a true variant here, since it builds no persistent index at all and pays for that choice in query economics; and Quickwit's format is at least Apache-licensed code, so the reader survives even if the company doesn't.)

So the three builds are: data and index inside the vendor's engine; data in your bucket with the vendor's index beside it; and data in your bucket in a table format that is itself the open standard. The middle build's marketing says "your data never leaves your account, no lock-in," and the first half of that sentence is true.

There's a second consequence of a closed index that I haven't seen discussed anywhere, and here I can bring measurements rather than inference. In my lab I ran the same queries against byte-identical Parquet through three open engines with pinned versions, and they disagreed: an allowlist filter usingNOT IN against a list containing a NULL returned zero rows on two engines, which is what SQL's three-valued logic says it must (any comparison against a NULL evaluates to UNKNOWN, so one NULL in the list voids the wholeNOT IN), and 80 rows on the third; a one-hour window query over UTC-stored timestamps returned the correct 600 rows on one engine and zero on two others under a non-UTC session (single machine, deterministic corpus, SDW Lab ocsf-temporal-null-coercion, Tier B). I could see those divergences because the data was in an open format and running a second engine against it costs nothing but compute. That check, answer equality across engines over identical bytes, is how I now gate any engine I add.

Now ask how you'd run that check against a platform whose index only its own engine can read, and you can't, because there's no independent engine to check its answers against. Whatever the query semantics of the vendor's engine are, with respect to NULLs, timezones, type coercion, or anything else, its answers are uncheckable against an independent implementation at any reasonable cost, and detection logic downstream inherits that on faith. I want to be careful here: I have no evidence that any of these vendors' engines return wrong answers, and that is precisely the point. The open-table build makes correctness a property you can verify, and the closed-index build makes it a property you have to trust. In security, where the query that returns silently-wrong rows is the one that costs you a breach finding, I think that difference belongs in the buying decision, and as far as I can tell nobody is putting it there.

Bottom line: the middle build's honest pitch is "S3 economics and your bytes in your account, in exchange for a searchability layer and a query semantics you take on trust."

If I were evaluating a platform in the middle build today, I'd ask one question before the demo:if you disappear tomorrow, what on my side can read what you've written into my bucket? If the answer is "the raw logs, and you'd rebuild the rest," that's not a deal-breaker, but it prices the exit, and the number should sit in the same spreadsheet as the ingest savings. I'd ask a second question right behind it: how would I check your engine's answers against an independent one?If the answer is that I can't, then the detection logic I build on top is trusting query semantics I've never seen tested, and I'd want that stated somewhere more binding than a demo.

And if you're the colleague who sent me down this path: the tip was better than its framing. There's no category called scanners, but there is an architecture that won, a middle build inside it that I haven't seen anyone fully price yet, and a correctness question sitting unasked in the gap between "your data" and "your capability."

Numbers and dates in this essay trace to the primary sources verified in the 2026-07-16 research sweep (Sequoia and SecurityWeek for the Series A; scanner.dev's own docs, pricing, and cost-comparison blog; AWS, Microsoft, Cribl, and Databricks primary announcements for the incumbent dates; HN Algolia for the traction counts; the SDW lab's cross-engine run for the answer-equality figures). The lock-in-moved argument and the cross-engine correctness question are my inferences, marked as such in the text.