Why modern data stacks haven't replaced SIEMs.

I want to set the cost argument aside deliberately, because it's well-trodden ground and I've made it at length elsewhere. The cost optimization paradox walks through why ingest-priced security data gets expensive in a particular way; the cost-to-serve at retention scale essay shows the index paying twice as data ages; and the write-endurance piece covers the storage-economics angle. If you want the dollars-and-terabytes case, those carry it. This essay is about something the cost math can't explain on its own, which is that even when the numbers clearly favor the move, the move mostly doesn't happen.

The case studies that get cited in these conversations are real. Huntress, running roughly three million endpoints, moved off Elasticsearch to ClickHouse and cut its bill by more than 90%; the dollar figures and the sourcing live on the Huntress teardown. The open table format underneath has corporate stewardship from Netflix, Apple, AWS, Snowflake and Databricks, so this isn't a fragile science project anymore. The technology is there. The reliability is there. And still, in most security organizations I've worked with, the SIEM stays.

So the explanation has to live somewhere other than the technology, and I think it lives in four organizational barriers that a benchmark never measures. None of them is irrational. Taken together they describe a security team weighing the risk of a transition against the risk of staying put, and frequently deciding, correctly given what they're accountable for, that staying put is the safer bet this quarter.

The second barrier gets described as a skills gap, and there is a genuine retraining problem in moving analysts from a proprietary search language to SQL against open tables. But framing it as a skills problem understates it, because the harder issue is that the three clocks involved run at different speeds. The technical payback on a platform migration tends to arrive over a year or more. Genuine team proficiency on the new tools takes the better part of a year to build. And management's patience for a transition that is visibly costing effort without yet showing results runs out a good deal sooner than either.

That ordering is the trap. A team can be doing everything right and still hit a stretch where the new stack isn't yet faster for them and the old one is being neglected, and that's exactly when an impatient stakeholder asks why detections feel slower than they did six months ago. The honest answer is that the team is mid-climb, but mid-climb is hard to defend in a status meeting.

What makes this worse for security specifically is that you can't pause operations to retrain. The SOC runs 24/7 whether or not the platform is in flux, so the team is learning a new query model while still carrying the full alert load on the old one. That double-duty period is where a lot of migrations quietly stall, not because the destination was wrong but because nobody budgeted for the friction of running two worlds at once.

The fourth barrier is the quietest, because it doesn't show up in any budget line. A team that has run the same SIEM for years has internalized its failure modes. They know which queries are slow and have learned to work around them, they know the dashboards that lie and the ones they trust, and they know who to escalate to when a particular ingest path falls over. Those known problems feel manageable precisely because they're known, even when an outside observer would call the platform objectively worse than the alternative. Familiar pain is legible; unfamiliar pain is frightening, and a migration trades the first kind for the second.

This is where the sunk-cost reasoning compounds the status-quo bias. Years of investment, a roster of certified analysts, and a body of institutional knowledge about the current system all weigh on the side of staying, and prospect theory explains why they weigh so heavily: Kahneman and Tversky's work shows that a potential loss looms larger psychologically than an equivalent gain, so the risk of the migration going wrong feels bigger than the cost savings it would unlock, even when the expected values say otherwise. A pure total-cost-of-ownership spreadsheet doesn't move a team across that gap, because the gap isn't arithmetic.

I want to resist the temptation to treat this as the irrational barrier, the one where the team is finally just being stubborn. It isn't. A team that deeply understands its current system's weaknesses genuinely is better-positioned to keep the lights on than a team that has just inherited a new system's unknown ones, at least for some transition window. The bias toward the known is a real and temporary operational advantage, and pretending otherwise is how migration pitches lose credibility with the people who'd actually have to run the result.

If the barriers are organizational, the things that overcome them are organizational too, and in practice they fall into a few recognizable shapes. The most common is a forcing function that takes the decision out of the team's hands. A budget shock, where the SIEM renewal lands at a number nobody will approve, converts an optional migration into a required one; Huntress's move off Elasticsearch followed exactly that pattern of costs reaching a level that forced the question. A compliance deadline can do the same, as can an executive mandate that simply decides the direction and removes the career-risk asymmetry from the team carrying it out, because once leadership owns the choice the individual architect is no longer the one who gets fired for it.

The greenfield case is the cleanest of all, because there's nothing to migrate. A new organization, a new business unit, or a security team standing up a function for the first time can adopt an open stack from inception and never pay the rebuild cost, which is why so many of the convincing modern-stack stories come from teams that never had a legacy SIEM to defend in the first place. The barriers I've described are barriers to migration specifically; they mostly evaporate when there's no incumbent.

For everyone else, the large majority with an entrenched SIEM, the move that addresses the most barriers at once is running both systems in parallel for a transition period rather than attempting a cutover. Parallel operation directly answers the risk aversion, because the old system stays live as a fallback the entire time; it answers the timeline mismatch, because analysts keep their familiar tools while the new stack is built up beside them and skills transfer gradually instead of overnight; and it lets the rule-rebuild happen incrementally, one detection at a time, validated against the incumbent's output before anyone trusts it alone. The cost of parallel running is that you're paying for two systems during the overlap, which is real and which is why the forcing function usually has to come first, but it converts a terrifying one-shot bet into a sequence of reversible steps, and reversibility is what a rational, risk-averse team needs before it will move at all.

None of this requires solving all four barriers simultaneously, which is the mistake most migration plans make. The workable path starts narrow, with one high-volume, low-detection data source where the cost savings are obvious and the detection risk is low, often something like DNS or proxy logs, and proves the model there before expanding. Validate the savings, validate the workflow fit, let the team build confidence on a corner of the problem where being wrong is cheap, and then widen. The barriers don't fall to a better argument; they fall to a smaller, reversible first step that makes the next one easier to take.

The modern data stack has won the technical argument for security data and is still losing the adoption argument in most enterprises, and the reconciliation of those two facts is that adoption was never decided on technical merit. It's decided on risk, timeline, rebuild cost, and the very human preference for a problem you understand over one you don't. The teams that keep the SIEM are doing something defensible inside their constraints, and a migration argument that treats them as laggards is arguing with the wrong premise.

What that means for anyone trying to make the case is that the work, with the destination mostly settled, is lowering each of the four barriers to getting there: giving the risk-averse buyer an accountable operator, giving the team a parallel path that keeps their fallback live, breaking the rule-rebuild into reversible increments, and accumulating the production proof that slowly drains the status-quo bias of its force. The cost case, which I've made at length elsewhere, is necessary and nowhere near sufficient.

The honest summary is that this is a story about organizations, not about engines. The benchmark was always going to be won by the open stack eventually. Whether and when a given security team adopts it turns on whether the transition can be made manageable enough that a rational person, accountable for keeping the detections running, is willing to take the first reversible step.