The thesis
Security data is a data-engineering problem.
The hunt you trimmed from ninety days to thirty because the query wouldn't finish. The log source that dropped at ingest and stayed dropped for a month. The detection that never fired and never said so. All three trace back to data-engineering debt rather than a security failure, and data engineering already paid most of that debt down.
The disagreement I take most seriously comes from Anton Chuvakin, who has written that "SIEM problems were never technology problems" and calls them people, process and "organizational physics" problems "wearing a technology costume" (Chuvakin, 2026-08-13; Tier B), and I think he's partly right, since a better data platform can rebuild whatever silos the organization already has, so I read the data-engineering fix as necessary but not sufficient.
Your team is closer than it looks. Every time an analyst pulls data out of the SIEM and wrangles it in tools they trust, that's the work; what's missing is the bridge. Require the open standards (Iceberg, OCSF, D3FEND, with Sigma carrying detections; Arrow comes along as implementation) in everything you buy and run, then adopt them one tool at a time. There's no bet-the-SOC migration: you fix the broken pieces, and each fix moves you toward the architecture. The MOAR Reference Stack is that architecture already assembled, so you can stand it up and watch it work.
I measured what the bridge buys instead of asserting it. On a 10M-event corpus, a schema-on-read index still wins the simple indexed lookups, and the lakehouse engines win the hunting-shaped aggregations threat hunters actually run by 5–62× (Tier B, single host, CV-gated, identical answers verified).
The three pillars
Three testable commitments
Each pillar is a structural commitment, a property the data platform has to demonstrate with evidence the program produces and updates over time.
Trustworthy
What this catches is the source feed that has been silently dropping events for three days while no dashboard alerts on it, the schema drift that makes half a detection rule's references resolve to nothing, and the asset database that is authoritative for ownership in 80% of cases, with a different source disagreeing on the remaining 20% and no one reconciling the two. The data is instrumented, validated, and lineage-traceable, so completeness, freshness, and schema conformance are measured per source and failures show up before analysts notice them in their queries.
The evidence is per-source data health reports running continuously, covering completeness, freshness, schema conformance, and OCSF conformance. (OCSF is the open standard for security data schemas; conformance means a given source's events match the shared shape rather than the vendor's proprietary format.) So the artifact is "we measured it over the last 30 days, and here is what we saw," in place of "the vendor told us their pipeline is reliable."
The lab has measured one of the quieter ways this fails. When a vendor's published field spec disagrees with what the product emits, a parser that follows the spec inherits the error, and in a positional format one omitted field shifted 5 of 12 fields to wrong-but-present values with nothing firing, while the same disagreement in key-value or JSON records came back as a single loud null (spec-vs-emitted-integrity, synthetic records, Tier B).
Well-connected
An analyst writes a hunt joining endpoint telemetry to the asset database and gets wrong results because the asset identifier field means three different things to three different tools, or the CMDB says 50,000 assets while the EDR sees 47,000 and the vulnerability scanner sees 52,000 and the delta gets papered over rather than named. Well-connected is the property that closes that gap: entities (assets, users, applications, configurations) resolve cleanly across sources, the data catalog knows which source is authoritative for which attribute with confidence and freshness scoring, and joins do what their JOIN clauses claim they do.
What evidence looks like is cross-tool gap analysis. When the sources disagree, the delta is documented, the authoritative source per attribute is determined with confidence and freshness scoring, and the coverage holes are explicit. Without that cross-tool view, calling the data well-connected is only an assertion.
The lab puts a number on the gap. On a synthetic 20,000-asset inventory observed through four tools, the best single tool, the CMDB, recovered 47.7% of the true asset state, a merge across tools that scored each reading for freshness and authority recovered 75.6%, and the remaining 24.4% was reported correctly by none of the four (ocsf-data-health, Tier B, synthetic). The assurance-gap essay walks through what that means for a hunt.
Performant
The data platform has to meet two distinct latency regimes on the same data. Sub-second detection and response (operational), and petabyte-scale historical threat hunting (analytical). The petabyte regime rests on the documented production deployments below (Cloudflare, Comcast, and Pinterest publish theirs), not on my lab, which measures at smaller scale. The single-engine assumption (one tool, one platform, accept the compromise on either side) is the regression that has been getting unbundled across the industry since 2022.
Most performance disappointment in security data comes from the spec being measured on a workload that doesn't match production, more often than from the vendor missing the spec. The brochure benchmark uses synthetic queries and idealized data shapes; the actual workload is dirty, skewed, and much of it is the same handful of queries running on different time ranges.
The evidence is reproducible benchmarks run against the actual workload, and measured that way, performant turns out to be two findings: the schema-on-read index wins the simple indexed lookups, and the lakehouse engines win the hunting-shaped aggregations by 5–62×, with 46.8× on the native engine and 10.1× over open-format Iceberg across the five-query average (Tier B, single host, 10M events, CV-gated, identical answers verified). The 9.0× blanket-ZSTD-22 compression figure (8.5× for Iceberg Parquet against raw JSONL) is a separate measurement from the same corpus and keeps its own label. The engine you land on inside the lakehouse tier is a separate decision that turns on catalog, concurrency, and operational cost, which is the trade-off the worked scorecard resolves against a named baseline. Methodology in the lab; reference implementation under NDA. Deeper read on what that means for petabyte-scale hunting.
Intellectual roots
The framing isn’t invented here. The first two pillars are the security reading of data-centricity, Dave McComb’s case for converging on one shared, extensible data model instead of letting every tool keep its own. Well-connected and trustworthy come straight from that single shared model. Performant is what security has to add: at petabyte volume the model has to be fast as well as coherent, which is why the platform here is a columnar lakehouse rather than a knowledge graph. The same lineage supplies the migration path. McComb’s incremental, stealth approach to retiring a legacy system is the same move as leaving a legacy SIEM one workload at a time.
Open standards
Open standards at every layer
Performant only stays performant if the layers stay swappable. What makes a layer swappable without re-platforming everything above it is an open standard at that layer. Five carry the weight, one per layer.
- Apache Arrow. The in-memory columnar format engines use to hand data to each other without serialization tax. Data layer.
- Apache Iceberg. The open table format that lets ClickHouse, Trino, Dremio, and Spark read the same files on object storage. Data layer.
- D3FEND. MITRE’s ontology of defensive techniques, the grounding layer that says what an observation means defensively, so you can ask which techniques you can even detect given what you collect. Ontology layer.
- OCSF. A shared event schema, so a source's events have one shape across tools rather than a per-vendor format. Schema layer. I treat it as a normalization-hygiene baseline that earns its keep, not as the schema everyone finally agrees on or a semantic backbone that fixes meaning, since conformance checks the shape of an event and not what its fields actually mean. ITU-T Study Group 17 is standardizing OCSF as Recommendation X.icd-schemas, and member states backed it for standardization in December 2025, but the draft text was still being revised through Study Group 17 as of this writing, so multilateral institutional backing for schema stability may follow but is not yet in hand.
- Sigma. Portable detection logic at the analysis layer. A Sigma rule compiles to ClickHouse SQL, Splunk SPL, or Sentinel KQL, so the detection content survives an engine change without a rewrite.
Sigma is on the list because of the analysis layer specifically. Open formats below it (Arrow, Iceberg) and an open schema (OCSF) make the data portable. Without an open detection standard, the detections themselves are still locked to whatever engine wrote them, and re-authoring a detection backlog is the migration cost that actually stalls re-platforming. Sigma's portability is real but bounded; correlation and stateful detections are less cleanly portable than single-event rules, so I treat the standards-portability claim as Tier B. The foundational-standards deep-dive works through each layer and where its standard holds.
The method
Empirical skepticism
Every claim I put in a recommendation is graded by the kind of evidence it rests on, from peer-reviewed research, ratified standards and named production deployments (Tier A), through practitioner talks and controlled first-party benchmarks (Tier B) and vendor-published material with its bias flagged (Tier C), down to speculation, which is labelled as such (Tier D). I don't reproduce the full rubric here. The tier definitions and the current grade on every active hypothesis live on the research page, next to the claims they actually grade.
The update rule is simple. Positions evolve when new evidence overturns them. That update history is kept publicly on /research as a "things I changed my mind on" log, twenty-nine contradictions documented and counting. I keep that log in public because a program that never updates its priors probably isn't testing them.
The full benchmark methodology, results, and reproducibility statement live on the lab page. Anyone can re-run it on their own workload.
If a future benchmark on a different workload reverses the result, I update, and the vendor neutrality you see on this site is a consequence of that skepticism, and I never set it as a goal of its own.
The four operating commitments
Compensation. No reseller margins. No commission on any product recommended.
Placement. No vendor-paid placements, and the matrix is scored under that disclosure.
Method. Benchmark spec, workload, and results are public on the lab; only the executable artifact is NDA-gated.
Review. An external review of the published results, published when it completes. A reviewer isn't confirmed yet, and they'll be named on disclosures once the review is done.
Foundation + three projects
A core validation program & three optional projects
The projects here are the depth tracks (the architecture and research work the thesis rests on). Before paid work paused, the foundation gate was offered as Data Health Validation and MOAR as MOAR Architecture Design, and the engagements page keeps that record. I'm not taking engagements while I work full-time, so the projects carry on as research and publications.
The foundation: data health, broadly defined.
A two-stage sequence that gates everything downstream. You don't ship detection content, hunting workflows, or machine-learning models on top of a foundation you haven't validated.
Stage one is per-source data health. Continuous reports establishing that what's in the lakehouse matches what each source promised, on completeness, freshness, schema conformance, and OCSF conformance.
Stage two is cross-tool gap analysis. When the asset database, the EDR, and the vulnerability scanner disagree about what's on the network, the delta is named, the authoritative source per attribute is determined, and the coverage holes are explicit. The assurance comes from that cross-tool view, because a program that produces clean per-source reports but never reconciles across tools cannot defend the claim that its connected dataset is complete.
Read the foundation deep-dive →Project 1
Research · FlagshipMOAR. Modular Open Architecture data infrastructure
The data platform itself. An Iceberg-based lakehouse, a vendor-neutral catalog (Polaris, Nessie, or Hive Metastore), purpose-fit query engines selected against the workload (ClickHouse, Trino, Dremio, or StarRocks), vendor-neutral routing (Tenzir, Vector, or Cribl), and the assembly that lets the same data serve sub-second detection and petabyte-scale hunting, where the petabyte end is proven by the published deployments on the references page rather than by my lab, whose own runs are single-node at up to a billion rows so far. The engine choice is scoped per table, though: each policy-protected table gets one engine and one catalog that own its writes and its access rules, because row- and column-level policy and audit identity don't yet carry reliably from one engine to the next. Snowflake's Jason Hughes and Jim Lebonitte lay that gap out in their July 2026 multi-engine lakehouse series (a vendor blog, and Snowflake gains from the single-engine advice), and the Iceberg REST spec only added row filters and column masks in September 2026, with no engine shipping them yet that I've found.
When to pick it. SIEM cost spiral, retention pain, multi-region or regulated query needs, or any infrastructure decision where vendor neutrality and open formats matter more than path-of-least-resistance.
Project 2
Research · DepthDetectFlow. Detection at thousands-of-rules scale, without the operational debt
Detection-as-code, CI/CD pipelines for detection content, telemetry feedback loops, automated regression testing. Each rule is a versioned, tested, deployable artifact whose performance and false-positive rate are continuously measured. What sets it apart is scale, since most detection programs cap out at hundreds of rules under operational debt, and DetectFlow is the discipline that lets a SOC carry thousands without collapsing under maintenance load.
When to pick it. Detection backlog growing faster than the team can maintain; analyst time consumed by tuning rather than hunting; incident retros showing detections that should have fired and didn't.
Project 3
Research · ~12–24 moMLOps-enabled model threat hunting.
Models surface anomalies and prioritize hunts; MLOps manages the model lifecycle (training, drift detection, retraining, evaluation) the same way DetectFlow manages the detection lifecycle. What I'm after is treating models as production artifacts with the discipline that detection content has earned, which is a narrower and more useful goal than putting "AI in the SOC."
Maturity: leading-edge. I haven't seen an incumbent doing this well at petabyte scale, though that is a read of the field rather than a lab-measured claim. Near-term posture is thought-leadership track first; service line follows once the foundation work has produced reference clients.
Reference architectures — in public
Working patterns published openly.
What gets discovered in the lab and in past client work gets distilled, anonymized and generalized, into reference architectures the rest of the community can use. Each architecture pairs the platform choices (lakehouse, catalog, engine, routing) with the matrix scores that justified them, and the failure modes that surfaced in production. The methodology is open, so you can reproduce each one against your own workload.
Two channels for distribution. First, through Security Data Works directly: the reference architectures catalog on this site, the matrix scoring that anchors each pattern, and, before paid work paused, the engagement deliverables that mapped the pattern to a client environment. Second, through the open standards themselves, where the method goes back upstream: pull requests to the D3FEND ontology, and the public ATLAS-to-D3FEND crosswalk with its audit trail.
The bet is that reference architectures compound as public goods. A pattern that lets one bank avoid a multi-million-dollar vendor lock-in lets a peer bank do the same; the methodology that surfaces a silent telemetry-loss regression in one SOC surfaces the same class of problem in another. The consulting incentive is to keep working architectures inside proprietary engagements, and publishing them instead is what the fair-broker model asks of me.
The takeaway
The takeaway is a requirement you write.
Write open table format, open schema, open detections and open ontology into everything you buy and run, then cross a measured, incremental bridge one tool at a time, because there’s no next platform to buy that does this for you.
A case for optimism: for the first time, every tool we need is already built and proven:
- the open formats
- the fast engines
- the shared schema
- the cheap storage underneath
We’re at an unusual moment, where the tools can finally meet some of security’s oldest needs, the ninety-day hunt that finishes and the dropped source someone notices, and what’s left is the will and imagination to apply them well.
Put the thesis to work on your data.
The research the thesis rests on is public, and so are the lab and the Matrix, so you can test it against your own environment. I'm not taking engagements while I work full-time, but I'm glad to talk about the research by email.