ETL vs ELT for security data: who owns the schema, and when.

ETL (Extract, Transform, Load) is the older pattern. You pull data out of a source system, apply a schema and any cleansing or detection logic in flight, and write the structured result into a destination store, so the schema is decided before the data lands and the destination store sees only what you chose to let through.

ELT (Extract, Load, Transform) inverts the last two steps. Pull data out of a source system, load it raw into a warehouse or lakehouse, and apply transformations later, usually via SQL or notebooks running inside the destination platform. The schema is decided after the data lands, and frequently decided multiple times by different teams running different queries against the same raw layer.

Stated that way it sounds like ELT is the more flexible pattern, and most "modern data stack" marketing implies as much, which is right for some workloads, though for security data specifically the ordering question collapses into a sharper one, namely where does the schema live, and who pays to maintain it?

If the schema lives at the source (inside a vendor's connector library, a SaaS integration, or a transformation step you don't directly control), you've handed that vendor a structural dependency. When their schema changes, your pipeline breaks. When their pricing changes, your migration is painful. If the schema lives in your warehouse or lakehouse, in version-controlled SQL or dbt models you can audit and rewrite, the dependency runs the other direction. The vendor becomes an interchangeable extraction layer, and the analytical logic stays yours.

The ETL pattern, modernized for security operations, looks like this: collect raw telemetry from EDR, firewall, DNS, identity, and cloud audit sources; run it through a streaming pipeline that applies detection logic, OCSF normalization, and sampling decisions in flight; and write only the surviving signals (plus a controlled sample of raw events) into the destination lakehouse.

Three vendors anchor the practical implementation choices. Cribl Stream is the commercial routing platform with the deepest source/destination integration library and a routing-by- value model: high-value events to the SIEM, everything else to the lake. Tenzir is the detection-focused open-core pipeline with native OCSF transformation and streaming ML hooks. Vector (from Datadog) is the open-source observability pipeline that's grown a security following because the configuration story is more legible than custom Flink jobs. Apache Flink and Apache Spark Structured Streaming sit underneath as the heavy-lift options when commercial platforms don't fit.

The cost shape inverts from ELT. You pay once during ingestion for the pipeline compute, and then almost nothing per query against the resulting lakehouse, because the lakehouse is storing 5-15% of the raw volume (signals plus sample), not 100% of it. Storage at $0.02-0.025 per gigabyte per month on S3 with Iceberg metadata is nearly free at that reduced footprint. ClickHouse or DuckDB on top of that storage handles analyst queries at a small fraction of Snowflake compute costs, because there's less data to scan.

The architectural cost is real, because you need a pipeline team (usually two to four data engineers depending on volume) and you need detection logic defined upfront, since the data you filtered out in flight is gone (or sampled, which is the same problem with statistics layered on top). So the tradeoff this essay is trying to make legible is that you give up query-time flexibility to get ingestion-time control, and you give up vendor compute lock-in to take on in-house data engineering investment instead.

ELT batch-query detection runs on a schedule, typically 5-15 minute intervals for the correlation rules that fire on log volume in any decent SIEM. Add the actual query latency (30 seconds to a few minutes against 900 TB of data with reasonable partitioning), and you arrive at total event-to-alert latency in the 5-20 minute range. That was the accepted baseline for the last decade of SIEM design.

ETL pipeline detection runs against the event stream as it arrives. Correlation logic executes per event (or per micro-batch in Flink terms), with latency measured in seconds. Total event-to-alert latency lands in the 1-10 second range for well-tuned streaming pipelines.

Three 2026 data points have made the 5-20 minute tier difficult to defend for detection-tier workloads specifically. CrowdStrike's 2026 Global Threat Report observed a 27-second fastest recorded breakout time (the interval from initial compromise to lateral movement) in their telemetry. A detection pipeline that needs 5-15 minutes to surface the initial event has already lost that race by more than 10x. Mandiant's M-Trends 2026 documented a negative mean time-to-exploit: on average, vulnerabilities are being exploited 7 days before patch release, which moves the defender's window from "time to patch" to "time to detect and contain." And Anthropic's Claude Mythos preview (April 2026, benchmarked that June) demonstrated economically viable autonomous vulnerability discovery and exploit generation at roughly $2,000 per kernel-class exploit (eight distinct exploits for $15,700 in API credits), with the AISLE rebuttal showing eight small open-weight models all reproducing its flagship FreeBSD exploit, meaning machine-speed offensive capability is no longer a frontier-lab artifact.

The honest read on those data points is that they don't argue that every security workload needs sub-second latency, but rather that the detection tier specifically has to operate at machine speed, and that scheduled-query detection at 5-15 minute cadence has shifted from a tunable operational tradeoff to a structural exposure. Hunting, BI dashboards, baseline computation, and post-incident forensics are still fine at minute-to-hour latency, because there an analyst is in the loop and the timing tolerance is different.

That's the latency case for pipeline detection, and the claim isn't that ETL is faster so ETL wins. The claim is narrower, that the detection tier specifically may no longer be a fit for batch-query architecture, and the implementation pattern that satisfies that constraint is in-flight processing, which happens to be the ETL shape.

1. Large scale where ELT compute escalates non-linearly

Above 5 TB/day, ELT compute costs cross into seven-figure-monthly territory at list rates. Even with negotiated discounts, the trajectory is uncomfortable. A 10-50 TB/day enterprise is paying for the same data to be scanned by the same detection rules every five minutes, and paying for it every time. Pipeline detection moves that compute upstream to a one-time-per-event cost. That's where the ETL economics start to dominate the conversation, and where the personnel cost of a data engineering team becomes a rounding error against the infrastructure savings.

2. Production operations with validated detection playbooks

Once detection rules are stable (meaning you know what you're looking for, you've validated precision above 90% and recall above 70% against historical data, and you have a maintained library of correlation logic) the case for running those rules in a pipeline strengthens. The exploration flexibility of ELT matters less when the analytical question is already known. Pipeline detection also satisfies the latency constraint the threat-model section makes explicit, which scheduled-query detection on an ELT platform may not.

3. Cost predictability and egress portability matter strategically

Pipeline compute is fixed per gigabyte ingested, which makes capacity planning legible. ELT compute is variable per query, which makes capacity planning a conversation with the vendor's sales team. For organizations where security budget is fixed and predictable (most of them) that predictability is valuable. Combined with storing data in your own S3 or ADLS bucket using open table formats (Iceberg or Delta), the ETL pattern preserves the option to swap query engines or storage platforms without paying egress, which is a strategic asset on a 5-year horizon.

1. "We need 100% raw data for investigations"

This is the most common objection to pipeline filtering, and it's usually false in practice, because the investigations that genuinely benefit from 100% raw data are a small minority while the ones that resolve with signals plus a 10-20% random sample of raw events are the large majority. The honest framing is to accept some bounded investigation limitation (5-10% of cases that would have benefited from full retention) in exchange for 80-90% cost reduction, and if the team can't accept that tradeoff explicitly, the ETL pattern is not the right fit and ELT is the more honest choice. The work is in making the tradeoff explicit rather than letting it stay buried.

2. "We'll migrate to ETL eventually, after we ingest everything for now"

This is the path that gets organizations stuck, because egress costs scale with the data volume already ingested, so twelve months of "we'll migrate later" can mean six- or seven-figure egress bills, plus the rewriting of the queries and the retraining of analysts on a new platform. The honest version is that if ETL is the eventual destination, you should start moving toward it within the first 6-12 months, before the migration cost compounds past the ELT savings it was supposed to capture, since the window where migration stays cheap is also the window where ELT still looks tolerable.

3. "We'll build a custom Flink pipeline from scratch"

Custom Apache Flink or Spark Structured Streaming is the right answer for a small number of organizations, typically those with existing data platform teams and very specific requirements that commercial pipelines don't fit. For most security teams the right starting point is a commercial or open-core platform (Cribl, Tenzir, Vector) for the first 12-24 months, with custom Flink reserved for components where the commercial option doesn't fit, because the economics of "build it ourselves" rarely survive contact with operational reality: the maintenance burden on a self-built streaming pipeline at 10 TB/day is substantial, and the time-to-production tends to run into quarters rather than weeks.

The ETL-versus-ELT debate looks like an ordering preference, but underneath it is a decision about where your schema lives, who pays to maintain it, and what happens when the vendor's pricing or product direction changes, which makes it a strategic decision dressed up as a technical one.

ELT on Databricks or Snowflake makes sense for small teams, early-stage detection programs, and exploration-heavy workloads. The cost shape is uncomfortable above 5 TB/day at list rates, and the egress lock-in is a real strategic exposure on a 3-5 year horizon.

ETL via Cribl, Tenzir, Vector, or custom Flink pipelines makes sense at production scale with validated detection logic. The cost story is materially better (80-90% infrastructure savings even after accounting for personnel) and the latency story is necessary, not optional, for detection-tier workloads in the 2026 threat environment.

Most production SOCs end up running a hybrid: pipeline detection for the validated rules, lakehouse queries for the analyst layer, and a smaller ELT platform retained for exploration and detection R&D. That's the architecture I recommend most often, because it puts each workload on the pattern that fits it.

The deeper point is that schema ownership compounds, so every quarter you spend with the schema living inside a vendor's compute platform is a quarter the vendor accumulates pricing power, while every quarter you spend with the schema living in version-controlled SQL against open table formats on your own object storage is a quarter you accumulate flexibility instead. That's an architectural claim rather than a technical one, and it's the claim I think most often gets missed in the ETL-versus-ELT framing.