Modular Open Architecture for security data.

The composition below isn't a default to adopt; it's what one set of constraints (single-node SOC scale, open formats, a small team) resolves to, and at other scales and shapes the layers move. With that said, an open architecture is only worth arguing for if it actually runs, and the reference implementation of this one does — one particular composition of those seven layers, which proves the architecture is tractable without proving it's the right composition for every shop: the reference stack writes one OCSF table to object storage and reads it back through five independent query engines (DuckDB, Trino, ClickHouse, StarRocks, and Dremio), across three catalog implementations (the Iceberg REST catalog, Nessie, and Lakekeeper) and two table formats (Iceberg and DuckLake), and a one-command check confirms every one of them returns the identical answer: 1,000 rows written in, 125 RDP connections found, the same count from every engine and every catalog. That agreement is worth verifying rather than assuming, because it doesn't come for free.

Nothing guarantees that agreement. Running the same correctness probe across thirteen Parquet readers on one 10-million-row file, two of them returned a wrong count with no error or warning while the other eleven read it correctly (chDB's newer reader and fastparquet), a fast and confident wrong number that for a security query is a missed detection or a bad compliance figure. That's more than one isolated engine and well short of most, so a wrong answer here is concentrated rather than universal, and the verification is what catches it cheaply. That's why answer-equality runs as a standing check in the stack instead of a one-time test, and why the verification is the part that earns its cost.

Run the same OCSF data into the open lakehouse and into an OpenSearch SIEM and the answers come back identical, but the lakehouse holds them in roughly a seventh of the space, 1.6 MB of columnar Parquet against the SIEM's 11.5 MB index at 200,000 events, the difference being the inverted index and the retained copy of the source the SIEM keeps. This is a single-host measurement, so read it as a footprint ratio rather than a cluster-scale promise, but the ratio is a property of the storage model and it holds at smaller scale too.

Normalization carries a cost too, and the stack measures it instead of assuming it doesn't. On a synthetic but controlled testbed, a coarse, schema-on-read normalization answers the routine SOC questions (counts, top-N, egress rollups) exactly as a fidelity-preserving store does, scoring a degradation of +0.000 on that routine set, on a scale where 0 means the coarse store answers it exactly and 1 means it can't answer at all. But on the adversary-relevant tail (beacon cadence, exact command lines, late-arriving events, identity closure across sources) the coarse store degrades by +0.188 on the de-gamed measurement — real MITRE APT29 telemetry with unmodified SigmaHQ rules, where the adversary tail's recall-loss runs roughly twice that of routine queries (0.35 against 0.16) — and by a larger +0.719 on the gameable synthetic testbed, because it physically discarded the evidence those queries need. The split is the finding: the questions that crater are the ones whose evidence the normalization dropped, and that falls out of the architecture, not the data volume.

Two more properties fall out of the open table format. Three writes produce three immutable Iceberg snapshots in an unbroken parent-to-child chain, and time-travel returns the exact state as of each one, the chain of custody a mutable SIEM index can't provide, at roughly the grade Reg SCI and SEC Rule 17a-4(f) ask for. And on a single host the engines answer the gated workloads in 4 to 14 milliseconds at a million rows, which says the architecture isn't paying for its openness in latency at this scale, though the cluster-scale crossover is a separate measurement the lab carries.

The product this serves is the Capability Matrix; everything above is the evidence behind its scores, run in the open on a stack you can stand up yourself. The lab carries the measured detail and the methodology, the evidence method explains how each claim is graded, and the generic four-phase roadmap is the reference sequence engagements build on. Every number here is single-host and first-party, and it's there to be re-run on your own hardware rather than taken on faith.

Retention economics flip.

Security data needs long retention for incident investigation and threat hunting. SIEM pricing punishes long retention; the per-GB-ingested model multiplies storage cost across the retention horizon. Lakehouse storage on object storage is cheap (S3 Standard at $0.023/GB-month, Glacier at a fraction of that), and query happens on demand against compressed columnar data. The per-query cost replaces the per-byte-stored cost. For a 5-year retention requirement, the economics aren't competitive; they're a different category.

OCSF-shaped data is portable across engines.

Once telemetry is normalized to OCSF (Open Cybersecurity Schema Framework, the multi-vendor schema standard for security data) and stored in Iceberg, multiple engines can query the same table simultaneously with no copy-out. The published benchmark validates this with four engines reading one Iceberg table on the same workload. The portability isn't theoretical; it's a property the lab verifies on every engine update.

Detection content survives engine changes.

Detection logic written in standard SQL ports cleanly between Trino, ClickHouse, StarRocks, and Dremio. SPL (Splunk's query language) and KQL (Microsoft Sentinel's) don't. The choice of detection-content language is a multi-year commitment that survives most other architecture decisions; standard SQL is the one that travels.

The SIEM doesn't have to die in the migration.

A common transition pattern: keep Splunk Search Head plus Enterprise Security for the SOC analyst experience; move historical retention and ad-hoc analytical workloads to Iceberg plus the chosen engine; federate. Splunk DB Connect is one bridge; native Iceberg integrations are emerging. Skill transition runs 6–18 months in parallel-run rather than a flag-day cutover. The Standard Chartered self-managed-SIEM teardown walks through one shape this transition takes.

Vendor swap-out is real, not a slogan.

Because each layer is bound by open formats (Parquet on disk, OCSF in schema, Iceberg in table format, OIDC in identity), replacing the query engine doesn't touch the data. Replacing the catalog doesn't touch the engine. This is the structural difference between modular and integrated. It's not free; the operational discipline (infrastructure-as-code, monitoring, runbooks) becomes mandatory rather than optional. But the optionality it produces is what justifies the operational work.

The migration assessment is designed to surface "no-go" answers before money is spent on implementation. Three patterns where the answer is reliably "stay where you are":

• Sub-1 TB total environments. The operational overhead of a six-component stack doesn't pay back at small scale. The SIEM economics that punish 50 TB/day workloads are still favorable at 200 GB/day.
• Pure compliance-checkbox shops. If no one queries the data after retention (the data sits to satisfy a regulator and never gets touched), dump to S3 cheap-tier and archive. MOAR's value comes from query performance and analytical workloads; for write-only retention, simpler is better.
• Teams without data-engineering capacity and no budget to acquire it. MOAR rewards operational discipline. Without it, the trade is one cost spiral for another, with the SIEM bill replaced by ops cost on a stack the team can't run.

The migration assessment surfaces these answers directly. A "no-go" finding is a successful engagement; under-scoping a deployment that lands the team in operational debt is the failure mode the framework is designed to avoid.