MOAR Architecture
You assemble it, you don't buy it.
Modular Open Architecture is a way of composing matured open standards rather than buying one vendor's platform:
- Iceberg for the table
- Arrow for the wire
- D3FEND for the ontology
- OCSF for the schema
- Sigma for the detections
- query engines you can change your mind about
Each is maintained in the open, each composes with the others over open formats, and any one part can be replaced without rebuilding the rest. That modularity is the safety mechanism, because it turns a re-platforming bet into an afternoon of pointing a new engine at the same tables.
The long-form argument, component by component, is the MOAR-explained essay. Below, the stack is organized the way you evaluate it: the three properties security data has to have, each naming the components that deliver it.
Trustworthy · pipeline & ingest, data quality & verification
What arrived is what happened, and it landed where you think it did.
Pipeline & ingest
Collect, parse, normalize, and route telemetry without losing or corrupting it, the layer where trust is won or lost before a detection ever runs. Storage lock-in mostly got solved by Iceberg and Delta, so the switching costs migrated up the stack here: every major platform vendor acquired, built, or bid on a pipeline capability in the eighteen months from early 2024 to mid-2025. The quieter problem underneath that vendor race is that the parsing boundary between vendors breaks first, on things as basic as timezones, and nobody in the chain is paid to own the fix.
Data quality & verification
A rule can compile clean and silently never fire because a value was wrong three layers down, and the same silent-failure mode shows up one layer below the detection logic, in the engines themselves. Reading one byte-identical Parquet file through twelve query engines, eleven matched the generator's ground truth and one returned a confidently wrong row count with no error raised, which is the case for treating answer-equality across engines as something you verify with a standing check, not something an open format hands you for free.
Well-connected · OCSF schema, D3FEND ontology, Sigma detections
You can follow one entity across every source that saw it.
D3FEND ontology
The grounding layer that says what an observation means defensively, so you can ask which techniques you can even detect given what you collect. It is where security meets the formal-ontology world, and where the honest finding is measured rather than asserted: the OCSF-to-D3FEND crosswalk is reciprocal at the class level, but the grounding underneath is roof-only, which leaves a real question about how much formal rigor a columnar lakehouse actually needs.
OCSF schema
One normalized event model so a detection written once fires across sources. The mapping labor is the real cost, and the silent-loss failure modes are the trap.
8 essays →Sigma detections
Vendor-neutral detection-as-code so the logic travels across engines. Atomic detections travel cleanly; correlation-heavy logic is not yet at parity.
3 essays →Performant · Iceberg + Arrow, query engine, catalog
You can ask the real question across the real window and get an answer before the investigation goes cold.
Table format & wire (Iceberg + Arrow)
The open table any engine can read off cheap object storage, and the columnar wire that moves data between tools without a parse tax at each hop. Lab tests at a billion rows found that the Parquet writer governs both read speed and file size, not the table format: byte-identical files read at parity across most queries, and at a matched codec PyArrow wrote 193 MB to DuckDB's 114 MB on the same data, so the usual claim that Iceberg is more storage-efficient than DuckLake turns out to be measuring the writer.
Query engine
Match the engine to the workload over the same Iceberg tables, swapping freely. One sharp caveat: an open format guarantees every engine can read the bytes, not that two engines compute the same answer.
7 essays →Catalog
The metadata layer that tracks which tables exist and who is allowed to see what, and the layer 2026's data-engineering consensus moved the contested ground to, partly because it is also where AI agents get their context. Governance there is real but uneven in practice: RBAC, row-level security, and branching exist, but enforcement is distributed across engines, delegated to features still in beta, and rarely portable from one catalog to the next.
Reference Stack
Run the architecture before you buy anything.
The whole architecture on this page ships as a runnable kit: a public reference implementation you can clone and run this afternoon, on synthetic data, with nothing to purchase and nothing to unwind. Assemble, don't construct.
What it is
The public reference implementation of the MOAR architecture.
The Reference Stack is the public reference implementation of the MOAR architecture: object storage, Iceberg tables, OCSF normalization, replaceable query engines, and a console — the UI inside the kit — that walks the full operator workflow. It runs on synthetic data with aggregate-only outputs and telemetry-injection guards. Those guards are honesty features, not demo shortcuts: what you watch is what the architecture does.
The performance question it answers is measured, not asserted: the lakehouse engines inside it beat a schema-on-read index by 5–62× on the hunting-shaped aggregations (46.8× native / 10.1× Iceberg five-query averages; Tier B, single host, 10M events, CV-gated, identical answers verified), and the index keeps winning the simple lookups, which I report just as plainly.
The operator spine
Six steps, end to end.
- 1
Setup — bring the stack up locally; one compose file, no cloud account.
Console screenshot · open full view → - 2
Configure — pick your sources and layer choices; the same decisions you'd make in your own environment, made visible.
Console screenshot · open full view → - 3
Land — synthetic telemetry lands as OCSF on Iceberg, open formats end to end.
Console screenshot · open full view → - 4
Gate — the data-health gate certifies what landed before anything downstream trusts it. This is the first broken-piece fix from the thesis, running live — see the .
Console screenshot · open full view → - 5
Analyze — multi-engine queries over the same open tables; swap the engine, keep the data.
Console screenshot · open full view → - 6
Migrate — the decision surface: what would move first in your environment, what stays, and what the kill-switches are.
Console screenshot · open full view →
Quickstart
Clone it and bring it up.
git clone https://github.com/flying-coyote/security-data-that-works
cd security-data-that-worksThe repository's first-run walkthrough takes it from there — one compose file, no cloud account, synthetic data only.
The console
The console that refuses to bluff a pass.
The MOAR console is a marimo app that runs the reference stack. You pick storage, a catalog, an ingest router, a query engine, and a schema standard, and it deploys that combination in Docker on your own machine. Before it calls the deployment healthy, a four-layer data-health gate checks source health, stack reachability, data quality, and cross-tool coverage, and it labels anything it has not actually measured as unmeasured rather than showing a pass it has not earned. A set of moar verbs then re-runs the stack's core claims against the live containers, swapping the store, the catalog, the table format, and the router in turn and confirming the answer does not move.
This is the operational counterpart to the rest of the stack. The component pages describe what each part is and what to read; this is the thing you run to prove the parts behave the way the architecture claims. It lives in the open at security-data-that-works, and every number in the gate and swap-proof tabs comes from its own evidence runs on a single host, Tier B.
There is a related idea with an overlapping name worth separating. The catalog-as-control-plane essay is an argument about an industry trend, that the Iceberg catalog is becoming the analytics control plane and the place AI agents get governed context, and where that argument holds for security versus where the real enforcement pushes back down to the engine and pipeline. That is the thing you read. The console here is the thing you run: it lets you compose the stack, catalog included, and then measures whether the composed stack behaves the way the architecture claims.
All of it is single-host, Tier B, on synthetic and structured corpora, so read it as evidence of mechanism and engine behavior rather than cluster-scale magnitudes. The deeper measured benchmarks are in the Lab.
The data-health gate
Four layers, and it says which ones it hasn't checked.
The gate is the console's claim to trust. It will not certify a deployment green until real machinery runs and returns a real result, and it holds three non-pass states apart that most dashboards quietly collapse into one. A check is unwired when no machinery exists for it yet, unmeasured when the machinery exists but has not run or could not run, and stale when a prior pass has aged past its one-day window and needs re-validating. None of those ever renders as a pass.
Layer 1: source health
Per-source freshness, schema conformance (no nulls in Iceberg-required fields), and completeness against a captured baseline. Honest about its own blind spot: it is a lake-side receipt, not a sensor-side capture, so a source that never sent and one dropped upstream look the same from here.
Layer 2: stack reachable
Observed rather than audited: pass if the catalog responds, fail if Docker is up but the catalog is not, unmeasured otherwise.
Layer 3: data-quality audit
Four measured checks: snapshot freshness, small-file compaction pressure, orphan files not referenced by the current snapshot, and schema conformance. Two further checks, a Parquet CRC bit-flip verification and a DuckLake tombstone-resurrection check, have no machinery yet and report unwired, never a pass.
Layer 4: cross-tool gap analysis
Coverage gaps between an authoritative inventory and the other tools' inventories, with a declared entity-resolution ladder that only closes a gap when the match confidence clears a threshold and the record is fresh. Ambiguous or stale matches stay gaps, because a wrong merge does not over-report a gap, it hides one.
The gate will permit a deploy on configuration integrity alone, since the data layers can only be measured after a stack is up, but it stays amber and says so plainly: deploy permitted, cannot certify green until the unproven layers run. That refusal to show green on faith is the whole point, and it is the same discipline the lab and the essays hold, expressed as running software instead of prose.
The failures nothing errors on.
The demonstrators are deterministic single-run showings of a mechanism, not measured magnitudes, and each catches a failure that raises no error on its own.
Flow reconciliation catches a silently dropped class.
A clean pipeline reconciled every OCSF class, 10,000 events emitted and 10,000 landed. A faulted pipeline with one class mis-mapped showed 4,000 emitted and 0 landed, the kind of silent drop a per-record quality check cannot see because the records that arrive are all fine.
Structure-aware queries catch absent-MFA logins that flattening hides.
Of 320 genuinely unprotected logins, 120 with an explicit false and 200 with the field simply absent, a naive flattened detection caught only the 120 and missed 200. The nested query that can tell absent from false caught all 320.
The commit tax shows up where streaming meets Iceberg.
On 100,000 rows, the same data written as one batch commit versus a hundred streaming commits moved metadata files from 4 to 301 and query planning from 9.9 to 142.5 milliseconds, while DuckLake's streaming planning stayed near 6 milliseconds. Which is the write-contract point the performant pillar makes, run rather than asserted.
Proving the swaps
Swap a layer, and the answer doesn't move.
Modularity is easy to assert and worth running, so the console runs it. Each verb shells out to the live containers and reports pass, fail, or blocked, and never a fabricated pass. On the 2026-06-20 evidence run, the same filtered query returned the same answer across every swap.
| Verb | Swaps | Result on the same OCSF batch |
|---|---|---|
| verify | the query engine | DuckDB, Trino, and ClickHouse each returned 1000 rows and a dst_port 3389 count of 125 |
| swap-store | MinIO vs SeaweedFS | both returned 125 |
| swap-catalog | iceberg-rest, Nessie, Lakekeeper | all three, independent implementations of the REST spec, returned 125 |
| swap-format | Iceberg vs DuckLake | both returned 125 over the same store |
| swap-router | Vector, Tenzir, Fluent Bit | all three emitted the identical OCSF Authentication record from the same raw Okta sample |
That swap-catalog leg is quiet evidence for a position argued elsewhere on the site: if the catalog can be swapped across three codebases without the answer moving, the catalog is not the layer carrying the durable lock-in. More verbs cover federated correlation, catalog governance and lineage, and the streaming commit tax; the console reports each against the running stack rather than a slide.
The boundary
What's public, what's premium, what's partner-mode.
Public
The whole operator spine on synthetic data, plus: the consultant-mode Matrix overlay, exercised against a bundled synthetic sample vault (the mechanism ships; the real vault never does); the field-mapping fidelity scoring method, so you can measure a source's OCSF mapping fidelity yourself; the Matrix scoring template and linter, empty — the structure with zero vendor values; and the free-mode shortlist generator on its no-scores fixture path.
Premium
The scored Matrix: per-vendor weighted totals, claim-versus-shipped deltas, measured per-vendor fidelity scores, workload-bundle recommendations. Delivered inside engagements, refreshed quarterly. The method is public; the numbers are the practice.
Partner mode
A third mode for delivery partners in the enablement track (never a paywall toggle): the per-customer recommender and the delivery surfaces a partner runs under their own SLA, with Security Data Works verifying the architecture.
Adoption →
Getting to the stack without open-heart surgery: tiered rule migration, the federated rollout playbook, and the cost-and-timeline reality.
Reference architectures →
Full-stack proof: named production deployments reconstructed from the public record, and the component and vendor blueprints underneath them.
Evidence & method →
The lab and the research notebook are one evidence-and-method surface. The lab runs and publishes the benchmarks, tier-labeled and reproducible; the research notebook turns those measurements into hypotheses, tracks where prior positions got revised, and keeps the method itself open to scrutiny. Both surfaces live at /lab and /research.
Two ways in.
If you build: clone it and run the spine end to end. If the gate or a query result surprises you, open an issue — the kit is the lab's front door, and re-runs are the point.
If you decide: the engagements page covers what a Modernization Discovery engagement looks at, what it costs, and what you get back.