MOAR, explained.

For a security reader who has spent a career inside a single search cluster, the modern data stack can sound like marketing, so it helps to name the four moves underneath it, because each one is a thing the ELK era did the hard way and the data-engineering world stopped doing the hard way.

The first move is separating storage from compute. In an Elasticsearch cluster the nodes that hold your data are the same nodes that answer your queries, so retention and query capacity scale together whether you want them to or not, and growing one means paying for the other. The modern stack puts the data in object storage (S3, ADLS, GCS) at roughly $0.023/GB/month, and spins compute up against it on demand, so you can hold a year of logs cheaply and only pay for the engine while a query runs. You can turn off every engine and the data is still there, readable by a different tool tomorrow.

The second move is the open table format. Object storage on its own is just a pile of files, and what makes that pile behave like a table (atomic commits, schema evolution, time travel, hidden partitioning) is a metadata layer written in the open and readable by any engine that implements the spec. That layer is Apache Iceberg, and it is the reason the pile of Parquet files in your bucket can be queried as one consistent table by Spark, Trino, ClickHouse, DuckDB, or StarRocks without each of them needing its own copy.

The third move is cheap, durable object storage as the foundation everything sits on, which is what lets the first two moves pay off, because storage that costs almost nothing per gigabyte is what makes long-retention security telemetry affordable to keep queryable instead of shipped to cold archive you can never search again.

The fourth move is query engines that got genuinely fast. Columnar engines like ClickHouse and StarRocks and embeddable ones like DuckDB closed the gap with proprietary indexed stores, so reading open Parquet off object storage is no longer the slow path it was a decade ago. Put those four together and you get the shape of the thing: open data the engines share, compute you scale on its own, and no single vendor holding the format your data lives in. The MOAR Stack is what you get when you take that shape and build it for security telemetry instead of for analytics dashboards.

Iceberg comes first because it is what the rest of the architecture organizes around, and it is the piece that the data-engineering world had already converged on by the time I went looking. In the ELK-era stack the equivalent layer was the search index itself, Elasticsearch's Lucene segments or Splunk's tsidx, and the problem with an index as your storage layer is that the index belongs to the engine, so your data and your query tool are the same decision, and changing one means re-ingesting everything into the other. Iceberg breaks that pairing by sitting as an open metadata layer between the query engine and object storage, so the table is described once, in the open, and any engine that implements the Iceberg spec can read it.

What that buys a security team is concrete. Schema evolution means that when OCSF adds a field or a cloud provider changes a CloudTrail structure, you update metadata rather than rewriting millions of Parquet files, so onboarding a new log source or absorbing a vendor format change stops being a downtime event. Time travel means a breach investigation can query the table as it existed at a past timestamp, which is how you reconstruct an attack timeline without restoring an archive. ACID transactions mean a query never reads a half-written file, so concurrent ingestion from many pipelines doesn't corrupt the count a compliance report depends on. Hidden partitioning means an analyst writes WHERE event_time > now() - interval '24 hours' and Iceberg applies the partition filter underneath, so the analyst doesn't need to know the physical layout to write an efficient query.

This is the layer your data physically lives in, which is exactly why an open, plural standard matters most here. Iceberg has first-class support across more than twenty query engines (Spark, Trino, Dremio, Flink, ClickHouse, DuckDB, StarRocks, Athena, Snowflake, BigQuery, and others), and the governance is plural: it's an Apache Software Foundation project with contributors across Netflix, Apple, AWS, and others, so no single vendor steers it. That convergence is recent and real, because Databricks acquired Tabular (the company founded by Iceberg's creators) in 2024, and Snowflake open-sourced the Polaris catalog to the ASF, which are the two clearest signals that the table format question has largely settled on Iceberg.

A fair caveat keeps this honest: Iceberg does not fit every workload. It's an analytic table format, not a real-time streaming sink and not a transactional database, and for a workload that only ever touches the last 24 to 48 hours the lakehouse benefits are thin. The detection tier that needs seconds-to-sub-second latency runs on streaming infrastructure (Kafka plus a stream processor) that feeds Iceberg rather than living inside it. Iceberg earns its place as the historical and investigative foundation, the layer that holds the long retention everything else reads from, and the architecture labels which workload each layer serves rather than pretending one layer serves all of them.

The first three components move and store bytes; D3FEND is the first one that's about what the bytes mean. MITRE D3FEND is a knowledge graph of defensive techniques and the digital artifacts they operate on, and its role in MOAR is to be the grounding layer, the published model that connects an observation in your data to a defensive technique and the artifact it concerns. The ELK-era equivalent was tribal knowledge: a senior analyst knew that a particular log line mattered because it touched a credential, or a process, or a network connection, and that knowledge lived in heads and runbooks rather than in a graph a machine could traverse.

What D3FEND changes is that the meaning becomes explicit and queryable rather than living in someone's head. When you can say that a given artifact in your telemetry maps to a digital artifact in D3FEND, and that artifact is operated on by a set of defensive techniques, you have a path from a raw observation to a defensive posture that doesn't depend on which analyst is on shift. That's the part of the stack that lets you ask structural questions: which techniques can I even detect given the artifacts I'm collecting, and where are the gaps where I'm collecting nothing that any defense reads. A mapping between an observation and a defensive technique is a claim about meaning, though, and claims about meaning carry a confidence that the architecture should show rather than hide. The discipline is to treat each cross-graph link as carrying its own support, so a multi-hop inference is only as trustworthy as its weakest edge, and an intent-blind inference is labeled as a lead rather than asserted as a fact.

The tribal-knowledge problem is exactly what a public, community-maintained ontology solves at this layer, because when the grounding is a published standard rather than a vendor's internal content library, the meaning your detections rest on isn't something you lose when you change platforms. D3FEND is the layer most foreign to a security architect coming from the SIEM world, since the old stack didn't really have an explicit ontology at all, it had product features and a content marketplace, and the shift to a published graph of techniques and artifacts is one of the genuine upgrades MOAR carries over the architecture it replaces.

Sigma is the detection layer, and it's the component that most directly attacks the lock-in a security team feels every day. In the ELK-era stack your detection logic was written in one vendor's query language, SPL for Splunk, EQL or KQL elsewhere, and that meant your accumulated detection content, the rules you'd tuned over years, was the thing most tightly bound to the platform, so changing platforms meant rewriting your detections rather than carrying them. That binding is a large part of why the four real barriers to leaving a SIEM (risk aversion, the skills gap, the migration gap, and status-quo bias) feel as heavy as they do: the rules are the institutional memory, and they don't move.

What Sigma adds is a vendor-neutral, YAML-based way of expressing a detection that compiles to many backends. You write the rule once against a stable field model, and a converter targets the query language of whichever engine or SIEM you're running, so the detection is portable in the way the data is portable. Sigma plus OCSF is the combination that does the real work: OCSF gives the rule a consistent field vocabulary to reference, and Sigma gives that rule a form that isn't married to one engine, so a detection can outlive a platform decision the same way the data does.

This is where reversibility either holds or collapses for the people who actually run the SOC, which is why an open, portable detection format matters more here than almost anywhere else in the stack. If you can leave a vendor but your detections can't come with you, the openness of the data layer is cold comfort, so a portable detection format is what makes the whole reversibility argument credible to a detection engineer rather than only to an architect. Sigma's coverage and the fidelity of its conversion to a given backend vary by backend and by rule complexity, and a portable rule still has to be validated on the engine it lands on rather than assumed to behave identically everywhere, so portable is not the same as free.

Lay the six components next to each other and the same property runs through all of them: each one is a maintained open standard that composes with the others over open formats, and each one can be replaced without rebuilding the rest. The table can change underneath the engine, the engine can change above the table, the schema can absorb a new source without touching the detections, and the detections can move to a new backend without moving the data. That is the whole reversibility argument, and it's a sentence a monolith vendor cannot say: if a layer choice turns out wrong, you swap the part and the data stays.

That reversibility is exactly what makes adoption safe and incremental rather than a bet-the-SOC migration, because the four real barriers to leaving a SIEM all soften when the move can be partial and reversible. You don't rip out the existing platform on day one; you stand up the lakehouse alongside it, land a couple of high-volume sources, query them from a familiar interface during the transition, and expand as the team's confidence and skills grow. Each step is reversible, so the migration risk is low at every phase, and the architecture is built to let an analyst keep working in the console they already know while the data foundation changes underneath.

One distinction is worth holding onto, because it's easy to confuse the durable idea with one instantiation of it. MOAR is the durable concept: maintained open-standard parts composing over open formats, any part swappable. A specific named stack (a particular store, catalog, engine, and router wired together at a stated scale) is one example of MOAR, the composition I happen to defend on a given host, and it is falsifiable on its own without putting the concept at risk. When you read a later chapter that names exact components, read it as one defensible instance rather than as the definition, because the architecture's value is in the composition pattern, not in any single parts list.

This chapter is the orientation, the map before the deeper walks, and each pillar here gets a chapter of its own that goes past the introduction: the table format choice and what changed with the latest Iceberg spec, the schema and the real cost of normalization, the detection layer and how far portability actually carries, the engines and the measured differences between them, and the migration path that ties it together into a sequence a team can actually run. The components are the vocabulary; the chapters that follow are where each one gets tested rather than introduced.