Security Data Works

The future of security data is already here; it’s just not evenly distributed.

1. Security at scale uses modern data architecture.

Teardowns 14Components 6Vendor blueprints 5Methodologies 4

At a glance

Named organizations already running it in production.

Security at scale already uses modern data architecture: open tables, replaceable engines, OCSF. Each card links to its verified primary source.

Security-nativeTier A
Orca Security
Petabyte-scale on a single Iceberg lake; 50%+ lower cost
Apache Icebergsource
Security vendor · OCSFTier A
Wiz
Findings exported as OCSF on Iceberg to your own lake
OCSF · Icebergsource
Internet scaleTier A
Cloudflare
Town Lake: one federated query across every source
Trino · Icebergsource
Internet scaleTier A
Comcast
10+ PB security data, >1 year hot retention
Snowflakesource
Security-nativeTier C
Panther
Detection-as-code SIEM; Databricks acquiring
Snowflake / Databrickssource
Security-nativeTier B
Huntress
3M endpoints · 16B events/day · ~$70K→~$5K/mo
ClickHousesource
Cloud scaleTier B
Palo Alto Networks
~10M events/sec · ~3 PB/day
Apache Beamsource
Security-nativeTier A
RunReveal
Ingest-time Sigma; ClickHouse's own security team runs it
ClickHouse · sigmalitesource
Regulated · financeTier B
DNB Bank
Norway's largest bank; Cyber Defense Center on Ibis
Ibis (multi-engine)source

Published architectures

2. They agree on key principles.

ArchitectureStorageQuery engineOpen table formatOCSFDetection‑as‑codeScale (verified)
Atlassian · Bsecurity lakehouse (public DAIS talk)Delta LakeSpark · Photon✓✓✓petabytes/day
DataBee · Comcast · Csecurity data fabricApache IcebergSnowflake · Databricks +1⇄✓✓✓10+ PB
Interpublic Group · BAmazon Security Lake, CISO-ledIceberg · Parquet–✓✓––
Panther · Databricks · Cdetection-as-code SIEM · Databricks acquisition pendingSnowflake · DatabricksSnowflake / Databricks per-deploy◑◑✓petabyte-scale
RunReveal · Cingest-time Sigma (sigmalite)ClickHouseClickHouse✕◑✓–
Wiz · COCSF findings exportParquet––✓––
DNB Bank · BCyber Defense Center · IbisApache IcebergIbis ⇄✓–◑millions/day
Orca Security · Aone transactional Iceberg lakeIceberg · S3Athena · Spark✓––petabyte-scale
Huntress · Bmanaged SIEM / EDRClickHouseClickHouse✕––16B events/day
Palo Alto · BCortex Data LakeKafka · BigQueryBeam SQL✕––10M/sec · 3 PB/day
Cloudflare · Afederated data platformIceberg (R2)Trino✓n/an/a1B+ events/sec
Netflix · AIceberg + Metacat catalogApache IcebergSpark✓n/an/ahundreds of PB
Uber · AHudi Merge-on-Read lakeApache HudiSpark · Flink✓n/an/a10 PB/day · 350 PB
Avride · Bmigrated Iceberg → ClickHouseClickHouse (was Iceberg)ClickHouse✕n/an/apetabyte-scale
LinkedIn · AOpenHouse · Iceberg control planeApache IcebergSpark · Trino +1⇄✓n/an/a3,500+ tables
Pinterest · AKafka · Flink · Iceberg on S3Apache IcebergSpark · Flink✓n/an/apetabyte-scale
Shopify · AIceberg on GCS via TrinoApache IcebergTrino✓n/an/a>1.5 PB single table
Robinhood · ACDC lakehouse · Hudi on S3Apache HudiTrino · Spark✓n/an/a10,000+ sources
Notion · Achose Hudi over IcebergApache HudiSpark · Snowflake✓n/an/a200B+ blocks
Apple · AIceberg at scale (VLDB 2024)Apache IcebergSpark✓n/an/atens of PB
How to read it
✓ uses it (in a primary)◑ partial / optional✕ engine-native (not an open format)– not statedn/a not applicable⇄ engine-swappable (stated)

Decoupled storage-and-compute isn’t a column here. Nearly every architecture already separates them, so it’s table stakes, and a column that’s ✓ for everyone tells you nothing. What actually varies, and what the columns show, is the storage format, the query engine, the schema standard, and the detection layer.

Open table format is where they split. The lakehouse camp (Iceberg, Hudi, Delta) and the engine-native camp (ClickHouse, Kafka) are both here and both at petabyte scale, so “everyone standardized on Iceberg” would be an easy claim to make and an easy one to disprove.

The query engine is the other axis that splits. Three camps show up: engine-native, where the store and the compute are one system (ClickHouse: RunReveal, Huntress, Avride); federated SQL run over the tables (Trino or Athena, used by Cloudflare, Shopify, Orca, Robinhood, LinkedIn); and an explicit portability layer, marked ⇄, whose point is that you can change engines, so DNB runs Ibis across DuckDB, Spark, and Snowflake, and DataBee runs the same Iceberg tables under Snowflake, Databricks, and Athena. That last camp is the “change your mind about the engine” claim, running in production.

OCSF and detection-as-code are the security-native platforms’ move. Atlassian, DataBee, Panther, RunReveal, Wiz state them; the hyperscale operators (Huntress, Palo Alto) built engine-native for raw throughput and don’t publish schema or detection standards, so those cells read –, meaning undocumented, not absent.

A convergence map of what they run, checked cell-by-cell against a primary, rather than a capability score.

>

Build an open stack that can prove itself

3. I made it easy to run them each yourself.

  • It’s free and open-source
  • Works on-prem/air-gapped
  • At petabyte scale
  • 10–47× faster than today’s SIEMs
  • If your SIEM searches S3, it can take load off of your SIEM today

Sounds too good to be true? I agree; that’s why this site has so much meat underneath it. I worked through building the evidence for it myself… it’s real, and you can use this work (all wide open and free) to prove it yourself.

Below: interactive setup steps from the MOAR console. Give it your constraints, pick tools, and notes will show how your picks interact given SDW testing.

Your constraints

Ingest· collect & shape

Why Vector →

Storage· object store

Why SeaweedFS →

Catalog· table metadata

Why Polaris →

Query· engine(s)

Why DataFusion →

Schema· the contract

Why OCSF →

Composed architecture

SourcesVectorSeaweedFSPolarisDataFusion

Schema contract across the flow: OCSF

How your picks interact

    Every cut and every note here links the benchmark or essay behind it: the same public method and candidate catalog from the Capability Matrix that an engagement starts from. This page is an eligibility pass, not a recommendation. What it doesn't do is weight your criteria and score the candidates against your workload; that instrument is what I bring inside an engagement. The working console ships in the open MOAR Reference Stack and runs against a live Docker stack you can stand up yourself, with its measurements in the Lab.

    Run the MOAR Reference Stack →See the engagement portfolioRead the matrix

    Start here

    New to the site? Here’s how it fits together.

    The reading order
    How the site's sections connectThree columns. Left: the Thesis states the claim. Middle: three evidence surfaces test it — the Matrix scores platforms, the Lab holds first-party benchmarks, and Research tracks open hypotheses and contradictions. Right: two depth surfaces — the Writing catalog's ten pillars and the MOAR reference architecture. Arrows flow from claim to evidence to depth, and a band beneath links to Engagements, where the method is applied to your environment.THE CLAIMTHE EVIDENCETHE DEPTHThesisthe argument, three propertiesMatrixthe scoring instrumentLabfirst-party benchmarksResearchopen hypotheses + contradictionsWriting80+ essays, ten pillarsMOAR Architecturethe reference designEngagements — the same method, applied to your environment

    Research

    Labs to re-run on your data.

    Zeek analytical workload · 10M events · 5-query average · single-node Tier B

    46.8× faster

    ClickHouse native vs. a generic schema-on-read SIEM, five-query average on an identical 10M-event workload — answer-equality verified. The 5–62× spread tracks query shape and realization, since the index wins the simple lookups while the lakehouse engines win the high-cardinality hunting aggregations where partition pruning and columnar scans pay off (46.8× native, 10.1× over open-format Iceberg on the five-query average). Single-node Tier B (32 GB RAM, 16 cores); full per-query breakdown and methodology in the lab.

    ClickHouse Native0.06 s
    ClickHouse + Iceberg0.28 s
    Schema-on-read SIEM2.85 s

    Reproducible Docker lab · methodology PDF public · full code and per-query data shared during engagement scoping

    Same data, same hardware, answer-equality checked, and every one of these is yours to re-run from the public Docker lab.

    Why now

    Why the SIEM model is breaking.

    01

    Attackers are faster than your detection cadence.

    Mandiant's M-Trends 2026 shows exploitation landing, on average, 7 days before patch release. CrowdStrike's 2026 Global Threat Report clocks the fastest recorded attacker breakout at 27 seconds.

    Three latency tiers →

    02

    Query performance has flipped.

    On a 10M-event Zeek workload, ClickHouse on its own storage averages 46.8× over a Splunk-style schema-on-read index across the five hunting-shaped queries threat hunters actually run, and still holds 10.1× reading the same data from open Iceberg tables, though the index wins the simple lookups. Same data, same hardware, same queries, answer-equality verified; single-node, Tier B, methodology in the lab. The architecture that index was sized for is gone.

    ClickHouse at petabyte scale →

    03

    Storage cost has flipped too.

    Object storage plus columnar formats (each field stored down its own column, so a query reads only the fields it touches) compress 8.5× against raw JSONL in our benchmark (Tier B; the exact ratio is corpus-dependent). Netflix, Huntress, and Insider run multi-petabyte data lakes at storage costs SIEM pricing can't reach. The trade gives up data freshness.

    The index pays twice →

    04

    Stream processing closes the freshness gap.

    Stream engines like Flink and RisingWave run detections on data in motion, pulling detection latency from the minutes-to-hours of a batch index toward sub-second for stateless rules, the tier that can run ahead of a 27-second breakout. The incumbents see it too: Splunk ships Federated Search for S3, expanded in Platform 10, because customers are already retaining telemetry in Parquet on object storage.

    Pipeline-based detection in stream processing →

    The five open standards

    Arrow: the columnar wire protocol

    Engine portability is only real if the driver is too.

    ADBC is a columnar-native update to JDBC (1992) and ODBC (1997), so you can swap query engines without rewriting the analyst’s tool stack. The cost it removes is the row-by-row serialization those old drivers do on every result set, which is why the gain concentrates on wide tables and large result sets.

    38× faster pulling one result set (DuckDB, TPC-H SF1; Tier B), and the SDW Lab measures 5–10× for ADBC over a JVM JDBC client on the same DuckDB query.

    Arrow and ADBC: the columnar wire protocol →
    >

    Start where your question is.

    ThesisThe argument

    The full case, and the three properties it holds every platform to, trustworthy, well-connected, and performant.

    3 pillars
    MatrixThe instrument

    The scoring method and the scored tables behind the evaluations, both public, with the applied per-environment read the part you hire me for.

    9 components
    LabThe evidence

    First-party benchmarks I ran myself, with the methods and raw results posted so you can check them.

    5–62×Tier B
    ResearchThe open notebook

    Working hypotheses and the contradictions against them, tracked in the open while the questions are still live.

    10 hypotheses26 contradictions
    WritingThe essay catalog

    Longer pieces sorted by topic, for when you would rather read the thinking in full.

    80+ essays10 pillars
    EngagementsWork with me

    If you want to put this to work, start here, from a scoped assessment through a full build.

    3 offerings$20K–$90K

    Ready to put the numbers to the test?

    Book a 30-min discovery call →Or stand up the Reference Stack first →

    Not ready to talk yet? Browse the engagement portfolio, or subscribe for new benchmarks and writing as they publish.