Handbook Chapter 1 / Thesis
The two cultures: evidence and marketing.
Data engineering and cybersecurity face nearly the same data problems, but they answer them from two different cultures. Data engineering decides by measurement it publishes in the open; security decides by the brochure it was handed. That difference, more than any single tool, is why security is a decade behind the data world it could have been learning from, and it is what the five principles below are meant to close.
Reading time: 12 minutes. Evidence tier: B (practitioner experience and the documented benchmark conventions of data engineering), with the volume and pricing figures flagged in the literature review treated as open questions rather than cited as fact.
Same problem, two answers
The walls are the same on both sides.
I came to this from the security side, at a data-engineering event in 2023 where I was the only person in the room who didn't already know what a modern data stack was, and I've told that story in full where it belongs, in the chapter on how the architecture came together and on the page on why this practice exists. What matters for this chapter is the thing I noticed once the surprise wore off, which is that the speaker was describing my job. The cost pressure, the scaling walls, the vendor lock-in, the struggle to run flexible analytics across messy sources: all of it was security operations, line for line, except this room had stopped treating any of it as unsolved.
The pressures really are shared, and they're worth naming plainly because the shape of the problem is what makes the borrowed answers transfer. Security telemetry volume grows year over year, fast enough that the bill grows with the threat surface rather than with anything you'd choose. The dominant SIEM is priced per gigabyte, so ingestion volume and cost move together whether or not the extra data earns its keep. The tooling is fragmented, a separate product for correlation and another for orchestration and another for threat intel and another for storage, each holding its own copy of the data under its own license. And analysts hit the same wall a data engineer hits, where query performance degrades as the dataset grows, so the most valuable questions, the ones that scan months of history, are the slowest and most expensive to ask.
I want to be careful with the numbers here, because this is exactly where it's easy to launder an estimate into a fact. The often-quoted figures for security data volume (a mid-sized enterprise generating a couple hundred gigabytes to a few terabytes a day) and for SIEM unit pricing (a few dollars per gigabyte per month) are directional at best. In the systematic literature review behind this work, the mid-market volume segmentation and the head-to-head SIEM pricing are both written down as gaps in the evidence, not as findings: validated production deployments exist at the very large end, but the mid-market numbers and the identical-workload price comparisons haven't been established. So I'm going to argue the shape of the problem qualitatively, because the shape is what's solid: volume rises, the bill rises with it, the tools multiply, and the queries slow down. The precise magnitudes are an open question, and I'd rather say that than borrow a confident number I can't stand behind.
The data-engineering world hit these same walls around 2015 and built an answer to them, in the open, and then moved on. Security mostly hasn't started, and the more I sat with why, the clearer it became that the gap runs through how each field decides what to buy, a difference of culture more than of catalog, not through which tools either one already owns.
The deeper gap
One field measures; the other reads the brochure.
Data engineering built a culture of evidence, and it shows in the everyday machinery of how the field argues. It benchmarks in public, with standardized suites anyone can run and read: TPC-H for ad-hoc decision-support queries, TPC-DS for the harder 99-query suite, and ClickBench comparing dozens of analytical engines on results anyone can reproduce. It shares architectures in conference talks and GitHub repositories rather than guarding them. And it writes up what failed, the migration that went sideways and the design that didn't hold, not only the wins. When a data engineer wants to know whether DuckDB or ClickHouse fits a workload, the move is to run the benchmark and read the numbers, and the numbers come with the query plans and the hardware spec attached so someone else can check them.
Security decides differently, and mostly on four inputs that are weaker than they look. The first is vendor marketing, the data sheet and the demo, where a "ten times faster than your SIEM" figure arrives without the workload, the dataset, or the hardware that would let you reproduce it. The second is analyst reports from firms whose business model creates a conflict of interest, where the most prominent ranking is qualitative and partly funded by the vendors it ranks. The third is anecdote, the war story from a peer's deployment, which is real signal but unquantified and rarely transferable to a different environment. And the fourth is the compliance checklist, which answers "are we allowed to run this" rather than "is this the right architecture," and which a field under audit pressure quietly lets stand in for the architectural question it was never designed to answer.
The contrast is sharpest in the room where the decision actually gets made. When a data engineer wants to know whether an engine is the bottleneck, they measure it. When a security architect wants to know whether their SIEM is the bottleneck, they read the vendor's data sheet. One of those is a measurement and the other is a brochure, and the trouble is that security has come to treat the brochure as if it were the measurement. A CISO comparing four platforms is looking at four faster-than-the-incumbent numbers, each produced by the vendor that benefits from it, none of them independently reproducible, so the most rigorous-looking figure in the room is whichever vendor most recently funded a study.
I should be honest that some of this is not security's fault, because the field is partly locked out of measuring by the contracts it operates under, where the standard SIEM agreement bars customers from running and publishing comparative benchmarks at all. That's a real structural cause and it deserves its own treatment, which I've given it in the long essay on running a benchmark that doesn't lie; for this chapter the point is the consequence, that when customers can't legally benchmark, the only security benchmarks that exist come from vendors, and a measurement culture can't form on top of a prohibition. And I'm not claiming data engineering is clean by comparison, because it has its own blind spots and its own vendor theater. But it built a habit of transparency around architecture decisions that security has never had, and that habit is the thing worth importing.
At a glance
How each culture answers the same question.
The same architectural question, asked in two fields, gets answered from two different evidence bases. Read the rows for the pattern; the caveats on volume and pricing are in the prose above and below, not in this table.
| When the question is… | Data engineering reaches for… | Security reaches for… |
|---|---|---|
| Is this engine fast enough? | A public benchmark on a standard suite (TPC-H, ClickBench), run on stated hardware, with query plans attached. | The vendor's data sheet figure, produced by the party that benefits from it, without the workload or hardware that would let you reproduce it. |
| How should I design this? | Conference talks, engineering blogs, and open repos where teams show the architecture and what broke. | A vendor reference architecture and a demo, plus a peer's war story shared privately in a hallway or a DM. |
| Which product do I pick? | Reproducible comparison plus documented trade-offs, weighted to the workload at hand. | An analyst ranking, partly vendor-funded and qualitative, treated as if it were a measurement. |
| Did this fail anywhere? | Written-up post-mortems; the failed migration is published alongside the successful one. | Mostly silence; failures stay inside the org, and the architecture lesson never compounds across the field. |
| Are we doing this right? | An architectural principle you can argue from first principles and test. | A compliance checklist that answers "are we allowed" and is let to stand in for "is this right." |
What actually transferred
Five principles, not a parts list.
When I went looking for what to bring back to security, the thing that transferred wasn't a tool, it was a short list of principles the modern data stack rests on, the ones that transformed data engineering roughly between 2015 and 2020. They're generic on purpose, because the value is in the pattern rather than in any single product, and I want to lay them out as architecture before any chapter that names exact components. Each one is ordinary in data engineering and each one cuts against the way security usually buys infrastructure, which is monolithic, proprietary, priced per gigabyte, and sold on a demo.
1. Separate storage from compute
In the old design the machines that hold your data are the same machines that answer your queries, so retention and query capacity scale together whether you want them to or not, and keeping a year of logs means paying for a year of query hardware you mostly aren't using. Separating the two puts the data in cheap durable object storage and spins compute up against it on demand, so you can hold long retention affordably and pay for an engine only while a query runs. For security that's the difference between archiving a quarter of telemetry to cold storage you can never search again and keeping it queryable for the investigation that needs it eight months later.
2. Schema-on-read over rigid schema-on-write
Security sources change constantly, because a cloud provider revises a log structure, a vendor adds a field, a new tool shows up with its own layout. A schema you have to define fully before you can write a byte turns every one of those changes into a rework project. Landing data first and applying structure when you read it lets you absorb a new or changed source without a downtime event, which matters for security precisely because the source you most need during an incident is often the one that changed last week. I've laid out that trade in the chapter on schema-on-read versus schema-on-write, because read-time flexibility isn't free and there are workloads where a fixed schema is the right call.
3. Modular components over a monolithic platform
The monolithic SIEM is one product doing storage and parsing and detection and search and reporting, which means a single weak part drags the whole thing and a single vendor decision binds all of them at once. Composing specialized components that interoperate lets you match each piece to its job and replace a piece that turns out wrong without rebuilding the rest. The cost of this approach is integration work, which is real and which I don't want to wave away, but the payoff is that getting one choice wrong stops meaning getting every choice wrong.
4. Open formats over proprietary lock-in
When your data lives in a proprietary format tied to one engine, your data and your query tool are the same decision, and changing one means re-ingesting everything into the other. Keeping the data in open formats that many engines can read, with an open table format like Apache Iceberg describing it, separates those two decisions and gives you an exit strategy that the vendor can't revoke. For a security team the open format is also what makes vendor competition real, because a vendor that knows you can leave behaves differently from one that knows you can't, and the openness is what gives the customer that power.
5. Evidence-based selection over vendor promises
This is the principle that ties the other four to the two-cultures argument, because the first four are about what you build and this one is about how you decide. Benchmark the options, validate the claims on your own data, and document the trade-offs, rather than accepting the vendor's number on faith. It's the import that matters most and it's also the hardest one for security to adopt, because the measurement layer is exactly the thing the field is contractually blocked from producing, which is why importing the principle requires building the venue for it rather than just deciding to be more rigorous.
From generic to security-specific
What the principles become when you build them for security.
The five principles are the generic modern-data-stack philosophy, and on their own they describe analytics dashboards as easily as they describe a SOC. The work of this practice is translating them into the concrete choices a security team actually faces, which is where they stop being a philosophy and become a specific architecture with named parts: an open table format as the organizing center, a shared event schema so detections aren't rewritten per source, a portable detection language so the rules outlive a platform decision, and workload-matched query engines you can swap without moving the data. That assembled set is what I call Modular Open Architecture, and the component-by-component build is its own chapter rather than this one.
One distinction is worth holding onto, because it's easy to confuse the durable idea with one instance of it. The architecture is the durable concept: maintained open-standard parts composing over open formats, any part swappable. A specific named stack wired together at a stated scale is one example of it, defensible on its own and falsifiable on its own, without putting the concept at risk if a particular parts list turns out wrong. When a later chapter names exact components, it's offering one defensible instance rather than the definition, because the value lives in the composition pattern and the evidence behind each choice.
The full walk-through of the components, what each one replaces from the older stack and why being able to swap any piece is what makes the architecture safe to adopt, is in the chapter that explains the architecture component by component. This chapter's job was the layer underneath that: why a security team should believe an evidence-based, open, modular approach in the first place, which comes down to noticing that the field it borrows from decides by measurement and the field it's trying to fix decides by marketing.
Why it's getting urgent
AI raises the cost of deciding on the brochure.
For a long time security could decide on marketing and absorb the cost, because a human analyst would eventually notice that the timestamp was off or that the useful field was buried, and would patch around it. The current push to put AI on top of security data takes that slack away, because an agent reasons over what it's handed and has no instinct that the data underneath it might be a lie, so a triage pipeline or a natural-language query layer is exactly as good as the foundation it sits on. A field that decides which platform to buy on a vendor's number, and never measures whether the data feeding the model is sound, is building the most demanding workload it has ever attempted on top of the input it has scrutinized least.
That's the deadline under the whole argument. The pressure security feels about AI is real, and it points, whether vendors say so or not, straight back at the evidence culture the field skipped, because you can't put an agent on a foundation you've never been allowed to measure and trust what it tells you. Importing the data-engineering habit of showing your work stops being a matter of taste at that point and becomes the precondition for the thing everyone is trying to build next.
The principles are the thesis. The evidence is the proof.
If this argument holds up for you, the next question is what the evidence culture looks like in practice — the benchmark methodology in the open, the artifact shared under NDA, the external review. That's the lab, and it's where the two-cultures argument stops being a claim and starts being a number you can check.