Independent benchmarks. Code in the open.

Every edge on the map except the cliffs gives warning, and each one has a signal you can watch for in your own estate, which is the honest way to use one-host coordinates: what travels is the ordering and the failure shape, not my absolute numbers. Seven questions you can answer from your own telemetry this week, one per dimension:

1. Scale — did your slowest routine query's p95 grow faster than your hot table's row count over the last two quarters?
2. Mix — is scheduled-cycle utilization (total scheduled-search runtime per cycle divided by cycle length) above roughly 0.7 and rising?
3. Concurrency — is busy-hour p95 more than about 3× off-hours p95, and growing with headcount?
4. Joins — does any correlation search die while its single-table components return in seconds? This is the one edge that gives no curve: it's a cliff, and the fix is a layout decision rather than a scale threshold.
5. Retention — what fraction of your data older than 90 days was actually queried last quarter, and at what hot-tier $/GB against an S3-class price?
6. Ingest — does source-to-queryable lag ever exceed your fastest scheduled detection's interval?
7. Sources — how many production sources have no owned, tested parser? The map draws this edge dashed, because it's practitioner evidence rather than a lab measurement.

I'd rather you measure these than take my coordinates. The public stack ships a runnable data-health gate (./moar healthcheck), and the public suites behind this map publish their methodology so you can point them at your own estate (the one NDA-held benchmark above is the exception, for the licensing reasons it documents). If measuring is itself the missing capacity, that is what the Data Health Validation engagement exists for — the done-for-you version of the same questions.

Ten million OCSF events, each engine's native store.

Five queries a SOC actually runs, from a full-scan rollup to a selective single-user time-window lookup, timed as medians after warmup. The answers matched on both engines for every one of them.

Every query returned identical rows on both engines, in both the query-the-file-in-place and native-store configurations, at one million events and at ten. That is the half of the interchangeability claim that has to be true, and it held: four of the five queries are byte-identical SQL across the two engines, and the fifth differs by a single token, DuckDB's // against ClickHouse's intDiv. The latencies traded by query shape and the gaps stayed modest, no wider than about 2.6×, with DuckDB generally ahead at this single-node scale and ClickHouse edging the time-bucket rollup inside its own MergeTree, and DuckDB loaded the ten-million-row corpus several times faster (457 ms against 3.9 s). So the swap is cheap and the answers come out the same.

What this is, and what it isn't.

The boundary matters as much as the result. This is embedded ClickHouse, not a tuned cluster, and a million to ten million rows sits below the scale where MergeTree's ordering and indexing and ClickHouse's distributed scale-out start to pay, and neither engine was hand-tuned. So this does not contradict the lab's other ClickHouse number — the flagship ClickHouse-server-vs-schema-on-read-SIEM result (46.8× on the five-query average, 21–62× on the hunting-shaped queries) is a different comparison at a different scale — and it does not lower ClickHouse's score for the dashboard-serving archetype in the matrix, which rests on exactly that large-scale latency, so the two results sit beside each other. As with the flattening benchmark this is Tier B, the corpus is synthetic, and the millisecond figures are medians on one machine; read them as which engine wins which query shape rather than as numbers to expect on your own hardware.

Both sources map into the class built for them, and both lose meaningful fields on the way. Okta's System Log lands 58% of its login fields on a typed OCSF attribute and loses the rest to coercion or the unmapped bag: a seven-value outcome enum narrows to three, the risk and threat signals sit in a free-form map OCSF has no typed home for, the forwarding chain flattens to a bare list of IPs, and the anonymizing-proxy flag has no endpoint attribute to land on. CrowdStrike's detections do better at 70%, because hashes and command lines are well modeled, but its ATT&CK tactic and technique arrive as flat strings where OCSF wants an id and a name both, its IOC type has to squeeze into a fixed enum, and its multi-action response bitmask collapses to one disposition.

The second finding is sharper, because it is about what a shipped mapper does rather than what the schema allows: OCSF 1.8.0 has a typed or coercible home for 36 of Okta's 50 login fields, but Okta's own open-source reference mapper carries only 18 of them into the event, leaving the autonomous-system, ISP, network-zone, and credential-type fields unmapped even though the schema would hold them. So the loss is partly what the standard cannot represent and partly what a shipped integration leaves behind, and that second gap is the larger of the two. Of eleven named detections scored against the result, eight lose at least one field they depend on, while three survive clean, including impossible-travel and known-bad-hash, because geo and file hashes map without a fight.

What this is, and what it isn't.

This is Tier B and it scores a documented mapping judgement against real vendor schemas and the real OCSF 1.8.0 schema, not production telemetry. Every OCSF target is validated against a checked-in copy of the schema, so a mapping cannot point at an attribute that does not exist, and the run is deterministic, scoring twice and asserting the two are identical before it writes anything. The Okta inventory is the login event family; the CrowdStrike inventory is the public Detection Summary Event, not the gated full Falcon Data Replicator schema, so the coverage figures are for those field sets, not for everything either vendor emits. The per-field calls are a judgement, published one line each so they can be argued with.

The headline is +0.72 on a scale where zero would mean coarsening hurt the adversary queries no more than the routine ones. It clears that bar because the routine control sits at exactly zero: the coarse store counts authentications, ranks destination ports, and rolls up egress bytes precisely as the fidelity store does, which is the point, because it isn’t a crippled strawman, it’s the store that serves the dashboards perfectly and then can’t reconstruct the breach. What it loses is specific. The 60-second beacon dissolves into 5-minute flow rollups. The base64 PowerShell payload is truncated away. The privilege escalation with no MFA looks identical to the 267 routine policy attaches because absence was written as false. The one human wearing an endpoint SID, an IAM principal, and an assumed role can no longer be closed into a single actor.

And not everything breaks, which is the honest shape of the result rather than a clean sweep: the kill-chain ordering and the dwell-time calculation mostly survive coarsening, because their evidence still reads through a five-minute rollup. The queries that crater are the ones whose evidence the coarse store physically threw away, and a benchmark that claimed every adversary query broke would be less believable, not more.

What this is, and what it isn't.

This is Tier B, and a first pass. The corpus is synthetic and the headline intrusion is one APT29-style chain on a single machine, so the magnitudes are specific to this testbed, not production rates. The coarse store’s every choice, from indexing on receipt time to rolling network into flows, flattening identity, truncating command lines, coercing missing booleans, and sampling rare DNS, is a documented real-world default written up alongside the code, and the routine control is the evidence it isn’t a strawman. The piece I can’t supply myself is the gate that would move this past a first pass: an independent practitioner confirming the coarse store resembles what shops actually build.

The one thing I did push on, because it’s the obvious objection to a synthetic result, is whether the gap is a property of coarse normalization or just an artifact of this one attack. So I ran a second, independent chain through the same frozen battery, a different actor on a different subnet with a different C2 domain and IP, a different encoded payload, and an SMB lateral move where the first chain used RDP, with the questions reading their indicators from ground truth rather than hardcoding them and the same background noise underneath so the chain itself was the only thing that changed. It landed at +0.71 against the first chain’s +0.72, with the same per-mechanism shape and the routine control still clean. The magnitudes are still specific to this testbed, but the mechanism reproduces across two unrelated attacks, which is the part I had half-expected to be fragile and wasn’t.

Putting the schema in front of the model is what moves the needle — invented paths drop from 0.99 to 0.69 — and the ontology-thinking discipline cuts them furthest, to 0.60. The control is the honest surprise: the conceptual grounding note lands at 0.72, exactly where a wrong-class grounding note lands, and neither beats schema-alone. So the grounding note’s content did no work here; it was extra tokens, and the reasoning discipline, not the grounding, is the lever. Exact path-correctness stays near zero throughout, because a 3.8-billion-parameter local model rarely lands the precisely-right path even with the schema in hand, which is why silent-error rate is the headline. That the formal grounding pays nothing over plain schema-conformance for field mapping is the same result the deductive grounding work found, now seen on a different task. Whether the discipline’s lift is a sub-frontier teaching effect or holds at the frontier is the next question — it needs the frontier leg this first pass doesn’t yet run.

On a 14-billion-parameter local model, the text-to-SQL baseline answers two of four adversary queries correctly — the single-table filters (the exact encoded payload, the privilege escalation with no MFA) — and fails the two that need more (the beaconing-cadence detection and the first-seen domain), where it writes SQL that errors or returns nothing. Result accuracy is 0.50 and the silent-error rate is zero: the failures are loud. That is the honest and slightly surprising first-pass observation, because the confidently-wrong tax that NL2KQL measured at ~42% in security telemetry is a frontier-model phenomenon, and a sub-frontier local model doesn’t reach it — it breaks visibly instead. Whether a formal rewrite avoids the silent errors a frontier LLM does make is the actual head-to-head, and it needs the frontier leg this first pass doesn’t yet run.

The OBDA arm is the formal counterpoint: Ontop exposes the store as a virtual RDF graph and rewrites SPARQL to SQL over an OWL2QL ontology. On the three adversary-tail queries OWL2QL can express, it is exact — three of three correct, zero silent errors. The five it can’t express (cadence, cross-source ordering, recursive identity closure, dwell, distinct-count — all aggregation or recursion, outside first-order rewriting) it refuses rather than guessing. That is the shape the head-to-head is built to surface: the formal rewrite answers fewer queries but is never silently wrong on the ones it answers, and its failures are a loud out-of-expressivity boundary, where a frontier LLM’s failures on this tail are silent. Which kind of failure you can live with is the real decision; the pending GraphRAG arm and a frontier text-to-SQL leg would complete it.

On the small-commit rung the file-write contract pays a per-commit metadata-and-manifest tax — roughly four files and about twice the commit latency of the transaction contract, and several times the storage. On large batches the gap narrows, because that per-commit overhead is paid once over many rows, which is the honest shape: the write contract matters for streaming, not for bulk load. The systems result underneath is the read-contract check — one engine reads both tiers and returns identical answers for the same logical data, so the unified-read-contract premise holds here for these two backends, once you write a version hint that DuckDB’s Iceberg reader needs and the writer doesn’t emit (a small, real interop wrinkle worth knowing about).

Reproducibility before performance.

A benchmark result that can't be re-run isn't a benchmark; it's an opinion. Every published result ships with the methodology document, the containerized environment definition, the data generators, the query suite, and the analyzed output. A practitioner with the same hardware and same data can re-run the experiment and verify the number independently. The reference implementation is shared under NDA with engagement prospects and qualifying reviewers — not published openly, because the comparison set includes commercial software whose licensing terms restrict third-party publication of comparative benchmark results. The methodology, the result, and the reasoning are public; the executable artifact is gated by a one-page NDA.

Identical workload across candidates.

Workload and queries are defined and pinned before any tool is run. No per-tool tuning advantage; the same query suite runs against the same data on every candidate engine. Vendor-recommended configurations are tested as additional rows in the result table — labeled clearly as vendor-recommended — rather than folded silently into the headline. The point is to characterize each tool's behavior on the workload, not to engineer the most flattering possible result for any one of them.

Documented caveats.

Every result ships with what was tested, what wasn't, and which workloads the result generalizes to. The benchmark is single-node; production environments typically run multi-node, and multi-node behavior is still outside what this suite captures. Single-node joins, though, are no longer an open question: the 2026-06 engine-join bench (Tier B, single host, 10M–60M-row tables over shared Iceberg) put StarRocks, both ClickHouse arms, and Trino on the same join suite, and while StarRocks led the join-heavy shapes and ClickHouse the most aggregation-shaped one, every engine answered every SOC join in under 1.5 seconds, which is why the lab reads engine choice at that scale as a catalog-maturity, concurrency, and operational-cost decision more than a join-latency one. TB-scale claims are a different regime that this bench neither confirms nor refutes. The benchmark covers analytical aggregation queries; full-text-search-dominated workloads aren't in the query suite, and the result generalizes less cleanly to those. The benchmark uses one log type (Zeek conn.log); other log shapes (endpoint telemetry, cloud control plane, identity events) carry their own performance characteristics. The caveats are the part that lets a reader know whether the number applies to their environment.

Vendor cooperation invited, not required.

Every vendor whose product appears in a benchmark is invited to review the methodology and propose configuration changes before publication. Vendor-proposed configurations are tested and reported as additional result rows, labeled clearly. The lab doesn't accept funded benchmarks, doesn't allow pre-publication vetoes, and doesn't allow vendors to dictate workload selection — but the methodology review is open, and that openness is part of why the published numbers survive contact with the vendors after release.

External review on annual cadence.

Once a year, an outside practitioner with the relevant standing — security data engineer, OCSF contributor, or analyst with quantitative-benchmark expertise — audits the lab's published results under NDA. They get the same access an engagement prospect gets: full methodology, full reference implementation, the underlying analysis JSON. Their signoff drives corrections to the public results. The first annual external review is scheduled for Q4 2026; the reviewer will be named here, with any flagged issues, when that review completes. Until then this is a forward commitment, not a claim of a signoff that already exists.