The cost optimization paradox.

Joe Reis's 2026 State of Data Engineering Survey asked 1,101 practitioners to name their biggest bottleneck, and compute cost landed near the very bottom of the list at 5.2%, ahead of only tool complexity, and far behind legacy systems and technical debt at 25.4%, a lack of leadership direction at 21.3%, poor upstream requirements at 18.8%, hiring at 11.4%, and data quality at 10.1%. The same survey found that where those practitioners actually spend their time is data modeling and transformation, at 55.4%, with ingestion and pipelines close behind at 48.1%, while compute cost barely registers as a thing they worry about.

It's a self-selected practitioner survey rather than a controlled sample, so I read the precise percentages as directional rather than exact, but the direction is unambiguous and it matches what I see on LinkedIn and in conference coverage: cost is not where the people building these systems are spending their attention or their worry. They are worried about the foundation, the data model and the upstream requirements and the legacy debt, and cost sits near the bottom of the list because it is the layer they feel they control least.

Meanwhile, the market the practitioners' employers are sitting inside is screaming the opposite. Redpoint Ventures' 2026 market update reports public software companies down roughly 20% year to date, the worst-performing S&P 500 sector, and public SaaS multiples sit at 4.1x forward revenue, the lowest in a decade. And 54% of enterprises surveyed are actively pursuing vendor consolidation, which is to say they're not "considering" it or "evaluating" it but already doing it.

That's the paradox, because the market is screaming "cut costs" while the practitioners are saying "let me build better models," and both groups think they're correct, which on their own terms they are.

The practitioners' priorities do not exist in isolation. They exist inside organizations that exist inside a software market that, in 2026, looks structurally different than it did in 2021 or 2022.

The SaaS repricing. Public software companies have lost roughly 20% of their value year to date, which looks less like a correction than a repricing of the entire sector, so when software multiples drop to 4.1x forward revenue (the lowest in a decade), most CIOs eventually receive a memo from finance that opens with "re-evaluate." This is a software-spend-broadly dynamic rather than anything specific to Splunk or to SIEM, and it captures every line item with a subscription attached.

The consolidation mandate. 54% of enterprises are actively consolidating vendors. That means contract renegotiations, platform migrations, and the elimination of overlapping tools. If you're running Splunk plus a data lake plus a SIEM plus a detection platform plus an XDR plus a DLP product, someone in finance is calculating the overlap. The conversation may not have reached your team yet, but the calculation is happening.

Databricks' March 2026 launch of Lakewatch (an agentic SIEM built on Delta Lake and Unity Catalog) accelerates this calculus. The launch messaging leads with an 80% TCO reduction claim relative to incumbent SIEMs. I want to be careful here: this is Tier D evidence (vendor launch claims, no independent production validation), and the 80% figure may not hold up across the workloads it gets applied to. What matters here isn't whether the number is right but that the launch is pitched at CISOs and CFOs rather than at the practitioners building the systems, because Databricks knows that the practitioner put cost near the bottom of their list while the executive put it at the top.

The hidden cost of fragmentation. Benjamin Rogojan documented a pattern in early 2026: "dashboard graveyards," where self-service BI created dozens of unmaintained dashboards, each built on semi-clean data from different sources. The data team moved fast, but nobody cleaned up. Every abandoned dashboard is a cost that doesn't appear in a line item but consumes compute, storage, and (most expensively) analyst attention when someone has to answer the "which number is right" question for a board deck. I treat this as Tier C evidence (practitioner observation, multiple LinkedIn captures), but the pattern matches what I see in the SOC analytics environments I work in. The security version is dashboard graveyards plus detection graveyards plus integration graveyards, all running quietly and all burning budget that nobody is auditing.

The market doesn't ask whether the practitioner thinks cost matters, because it produces a renegotiation cycle, a consolidation wave, and a fragmentation tax regardless, and someone has to absorb them, which is usually the architecture that was designed without those forces in mind.

If you're building or evolving a security data platform in 2026, the paradox has three concrete implications. None of them require taking sides between the practitioner ranking and the executive ranking. They require designing for both clocks at once.

1. Architecture must serve two masters.

Design for capability and cost from day one. Open formats and open engines do this naturally: Iceberg or Delta at the table layer, Parquet on disk, OCSF at the schema layer, Sigma at the detection layer, with ClickHouse, DuckDB, or Spark as engines you can swap. That stack gives the practitioner the modeling flexibility and analyst-accessibility they actually want, and it gives finance the portability they will eventually demand when the consolidation memo arrives. One caveat I'd add from my own lab, because it's the kind of thing that doesn't surface until you check for it: those engines are interchangeable for the SQL and roughly for the latency, but not unconditionally for the answers — I had one of them return a filtered count tens of rows short of the others over the identical Parquet with no error raised, so the swap is only as safe as the cross-engine answer-equality check you run behind it (that case is its own essay).

Proprietary schemas (CIM, ECS) and vendor-locked storage (Splunk Cloud, Sentinel-managed storage) optimize for one master only, usually the master who signed the original contract. When the other master arrives with a budget cut, the architecture is stuck. The cost of unwinding it tends to be larger than the cost of building it twice the right way the first time. I cover the migration math in detail in the hidden cost of SIEM migration, which lays out why "we'll just migrate later" rarely survives contact with the pipeline-rewrite bill.

2. Cost savings are a side effect, not the goal.

Don't lead with "we'll save 50 to 80%," because practitioners put compute cost at 5.2% and won't be moved by it, so lead instead with better data modeling, better detection, better governance, and a better analyst experience. The cost savings follow naturally from the architecture choices (open formats, commodity storage, decoupled compute, no per-GB SIEM ingestion tax), and when the CFO asks about cost six months later, the numbers will be there. But the architecture was built for capability, not for cost, which means it actually works, unlike the "cost-optimized" architectures that cut corners on data quality and discover three quarters later that the detection engineer can't write a working rule against the data shape the cost-optimization decision produced.

This is also the honest framing of why the SIEM cost crisis is genuinely a crisis without it being a vendor-bashing story. Splunk, QRadar, Sentinel, and the rest priced themselves for an era when security data volume was measured in gigabytes per day. The volumes are now measured in terabytes, and the pricing models did not bend gracefully, which is a problem of pricing-model drag rather than a problem of bad vendors. The fix isn't to call vendors greedy but to design an architecture where the ingestion volume and the analyst-facing query volume don't have to share a pricing surface.

3. Build for the consolidation wave.

The 54% consolidation figure isn't a one-time event but a structural shift that will repeat, so design your architecture assuming at least one of your current vendors will be eliminated, acquired, or repriced within 18 months. The list below is not a checklist for ideological openness; it's a checklist for surviving a consolidation cycle without rebuilding from scratch:

• Schema portability. OCSF, not CIM. If the schema is owned by a single vendor, the consolidation cycle can take it from you.
• Storage independence. Iceberg on S3 (or Delta on S3), not vendor-managed storage that you can't read without the vendor's compute. See Iceberg vs Delta Lake for security data for the table-format trade-offs.
• Query engine flexibility. ClickHouse or DuckDB or Trino or Spark, but not locked to one. If swapping the engine is a Python-library change rather than a data-migration change, you've designed for the consolidation horizon. I put a number under that swap in the lab: DuckDB and embedded ClickHouse returned identical answers to the same OCSF queries, so the SQL and the results port, and the latency cost of moving between them was modest and tracked the query shape rather than amounting to a rewrite.
• Detection logic portability. Sigma rules, not SPL. If your detection logic is written in a vendor's proprietary query language, the consolidation cycle takes your detection content too.
• Pipeline portability. The ingestion layer is where vendor lock-in tends to be most invisible. I cover the specifics in pipeline lock-in: the short version is that pipeline tooling deserves the same portability scrutiny as storage and compute.

None of these moves require believing that any specific vendor will fail or be acquired. They require believing that consolidation cycles happen, that they affect roughly half of enterprises in a given window, and that the cost of preparing for them is smaller than the cost of being surprised by them. The 54% figure is the operative one. Build to survive being on the wrong side of it.

If you're a data engineer or security architect reading this and thinking "I don't care about cost, I care about getting the modeling and the governance right": you're not wrong. Those are the things that determine whether the system actually works. The practitioners who put cost near the bottom are correctly identifying that the system has to function before it can be cheap.

But add one item to the list, which is portability, and not because it saves money today or makes the dashboards prettier, but because when the consolidation mandate arrives (and the 54% number says it's already arriving for half the room) you want your work to survive the platform change. Portability is the thing that lets the capability investment compound across consolidation cycles instead of being rebuilt every eighteen months.

The portability moves are mostly free at design time and expensive at retrofit time. Choosing OCSF over CIM costs roughly the same engineering hours either way at greenfield, and saves a schema-rewrite project later. Writing detection logic in Sigma costs roughly the same as writing it in SPL at greenfield, and saves a detection-content port later. Choosing Iceberg or Delta over a proprietary table format costs roughly the same at greenfield, and saves a petabyte-reprocessing bill later. The "build it twice" cost only shows up if portability wasn't a first-class concern at design time, which is the default outcome when cost is ranked near the bottom.

The smart move isn't to flip the ranking but to add portability quietly, treating it as a property of the capability-first design rather than a separate cost-first design, and letting the eventual consolidation-cycle savings happen as a side effect. The practitioners who do this are right about their priorities and prepared for someone else's priorities at the same time.

The practitioners who put compute cost at 5.2% are not careless about money. They're accurately reporting that cost is not the layer of the system they control. Their priorities (data modeling, AI-native pipelines, governance) are the priorities that determine whether the system works in the first place. Without those, no amount of cost optimization produces a defensible security program. The capability layer is the binding constraint, and they correctly identified it.

The market doesn't care about that ranking. The SaaS repricing, the consolidation mandate, and the fragmentation tax produce a cost optimization wave whether the practitioner asked for one or not. 54% of enterprises are pursuing it actively today, and the cycle repeats. The architecture either survives the wave or gets rebuilt under it.

The resolution isn't to flip the practitioner ranking. The resolution is to design capability-first architectures that happen to be portable (open table formats, open schemas, open detection languages, swappable query engines) so the consolidation cycle takes a contract instead of an architecture. Cost optimization, on that design, is a side effect of having built the right thing, which is the version the practitioner ranking implied all along, even if the survey didn't ask the question that way.

The honest framing of the SIEM cost crisis is that it's a pricing-model-drag problem layered onto a consolidation cycle, not a story about bad vendors or careless practitioners. The crisis resolves when the architecture stops asking the ingestion layer and the analyst layer to share a pricing surface, and starts treating cost optimization as a property of portability rather than a property of vendor selection. That's a multi-year project, and it survives whichever way the next consolidation wave breaks.