I was the only one in the room with my hand down.

The honest answer, the one it took me a couple of years of digging to be sure of, is that the security practitioner mostly can't show their work, and three structural things keep it that way.

The first is that they're locked out of measurement. Splunk's General Terms, Section 1.2(v), tell every customer they agree not to use the product to “analyze, test, characterize, inspect, or monitor its availability, performance, or functionality for competitive purposes.” Read that plainly — a security team that wants to know whether ClickHouse or Trino could run their analytical workload faster, on their own data, on their own hardware, is contractually barred from finding out. It isn't a Splunk quirk. Snowflake, Databricks, and Oracle carry comparable language, and Oracle's traces back to the 1980s DeWitt clause, named for the database researcher Oracle stopped from publishing comparative results. The consequence compounds, because when customers can't legally benchmark, the only security benchmarks that exist come from vendors whose own products don't restrict them, so every “10× faster than your SIEM” figure you've read was funded, by structural design, by a party that comes out ahead in it. The architects who would gain the most from independent measurement are precisely the ones forbidden to produce it.

Sit in the room where SIEM modernization actually gets decided and you can watch what that produces. A CISO is comparing four platforms, each with a faster-than-the-incumbent number on its data sheet, each produced by the vendor that benefits from it, and none of them independently reproducible, because the customer who could reproduce them is bound by the clause that prohibits exactly that. The most rigorous-looking figure in the room is whichever vendor most recently funded a study, and the decision gets made on the measurement that happens to be available rather than the measurement that would be useful. I've written the long version of this on the economics of independent measurement; the short version is that the measurement layer security needs structurally cannot exist under the contracts security operates under.

The fix isn't to open-source a benchmark and tell everyone to run it, because the moment the comparison set includes commercial software under a restrictive license, shipping a turn-key reference implementation puts either the publisher or the downloader in breach. What works is the shape data engineering already settled on with TPC-H, TPC-DS, and MLPerf: publish the methodology, the hardware spec, the query suite, and the results in the open; share the executable artifact under a one-page mutual NDA with qualified reviewers; and put a named external reviewer on the methodology each year, with their signoff published. Security has been waiting for that layer for a decade, and the contract regime is most of why it never arrived.

The second is that the knowledge is tribal, and nowhere does that show more clearly than in the parsing layer that turns a vendor's raw logs into fields a SIEM can search. That layer is owned by no one in the chain, and it breaks where no one is looking. PAN-OS firewall syslog ships with no timezone value at all, and the tribal knowledge is that you set every firewall to UTC so the assumption downstream holds. That works until someone stands up a new pair of firewalls on local time, and I once watched a regulated utility end up with an 18-hour spread across its estate, a fabricated incident timeline that still rendered green on every dashboard. In 2023 I submitted a pull request to Palo Alto's own community Splunk app fixing well over a hundred broken and missing field extractions; it was never merged, and the repository has since been archived. I don't read that as malice, I read it as incentives: nobody was paid to merge it, the data being a little wrong in the SIEM isn't a problem the vendor experiences in its own console, and the integrators who know where every vendor hides the bodies are paid to keep that knowledge private. It isn't only Palo Alto, either; Splunk stamps Zeek logs with the wrong timestamp by default, taking the write time instead of the event-start time security actually needs, and Tenable's most useful vulnerability context arrives buried in an unparsed nested field, present and useless at the same time, and I hit and fixed both, so the shape is structural rather than one vendor's neglect. The parsing layer nobody owns is the full version of that story.

The third is that there's no forum where the architecture conversation even happens for security. The structures we do have for sharing, the CISA Joint Cyber Defense Collaborative, the ISACs, the Chatham-House-rule groups, are organized around threat intelligence and incident response, and they're genuinely good at that, and they matter. But they aren't built around data architecture, so a security architect who wants to compare notes on lakehouse design or OCSF mapping has no equivalent of a Subsurface or a Data Council to walk into. Data engineering talks about its architecture in public; security talks about its adversaries in private, which is the right instinct for indicators of compromise and the wrong one for infrastructure decisions, and the result is that the architecture lessons never compound across the industry the way the threat intel does.

I ended up doing something about that gap in a small way, by convening a periodic group of security-data architects who compare notes on lakehouse design, OCSF mapping, and the internals of formats like Iceberg and DuckLake, because the venue I wanted to attend didn't exist and the only way to get one was to start it. It's a peer forum and not a sales channel, and the fact that it had to be built from nothing is telling: data engineering has Subsurface and Data Council and a dozen others, and security's architects have been improvising in direct messages and hallway conversations.

The fix needs someone who is both a data engineer and a security practitioner, which is a strange and slightly unreasonable thing to ask for, because each of those is hard on its own and the combination is rare. The data scientist who's never run a SOC doesn't know that an alert's timestamp is a claim that has to be earned, or that the “user” field means three different things in three different tools. The security expert who's never built a pipeline doesn't know that storage and compute can scale separately, or that an open table format lets you change engines without re-ingesting a byte. Each side has a blind spot exactly where the other has a reflex, so the person who carries both can read a board that neither half can read alone. Why would anyone sign up for that? Because the bridge between them is exactly where the value sits, and because the crossing has already started, on its own, from the security side. Threat hunters are turning into data scientists without quite noticing it happen: they're living in Jupyter notebooks, building features over telemetry, versioning detections as code, reaching for MLflow to make a hunt reproducible six months later. I've mapped that path in the work on MLOps for threat hunters and moving from notebooks to reproducible artifacts. The honest move is to name the trend and lean into it, to treat the hunter who's learning to think like a data scientist as the future of the role rather than a curiosity, because that person is the one who can finally tell security's data story in a language the rest of the data world already speaks.

All of which is why I built the practice the way I did, around evidence rather than opinion. Security Data Works exists to make public the things the rest of the chain has reason to keep to itself. The benchmark methodology is open on the lab page, the reference implementation is shared under a one-page mutual NDA so it can include commercial software without putting anyone in breach, and an external reviewer is named each year with their signoff published, so a number I publish is something you can check rather than something you have to trust. The first number through that process is a plain one: on a ten-million-event Zeek workload, on identical hardware, an open columnar engine ran a five-query security analytical suite 46.8 times faster on average than the dominant schema-on-read SIEM, with the margin running 21 to 62× on the hunting-shaped aggregations and the SIEM's index winning the simple lookups, answer-equality verified on a single node, with 8.2× compression, and the methodology and caveats are published so you can argue with it line by line. The capability matrix scores the components of an open architecture against that same evidence, and it scores against the disclosure rather than around it, so an active partnership with one vendor doesn't move a competitor's score. The four operating commitments behind all of it are dull on purpose: no reseller margins, no vendor-paid placements, methodology public with only the executable artifact gated, and an annual external review. None of that is glamorous, and none of it resolves the structural problem overnight, but it is what standing in the gap requires rather than just describing it.