Technology deep-dive

Pipeline-based detection in stream processing.

Pipeline-based detection means the detection rule fires while the event is still moving through the stream processor, rather than later, when an analyst or scheduler queries data that has already landed in a SIEM index or a lakehouse table. That single shift (detection at ingestion time rather than at query time) changes both the latency profile and the failure modes, because latency moves from minutes-to-hours down toward sub-second in the happy path, and the failure modes move from "did the query run on schedule" to "did the watermark advance correctly, and is the state store sized for the join history I'm asking it to remember."

Reading time: about 22 minutes. Evidence tier: B overall (streaming framework documentation, practitioner blog posts, vendor case studies). Cost numbers in this essay come from public vendor pricing applied to an illustrative 10 TB/day workload. Treat them as a starting hypothesis for your own modeling, not a quoted figure for any specific deployment.

Stream processing has its own vocabulary, and the words matter because each one names a place a detection can go wrong. Four terms come up repeatedly when discussing pipeline-based detection, and I translate each one here in security-operational language rather than data-engineering shorthand.

Windowing

A window is a finite slice of an otherwise unbounded event stream. "Five failed logins in ten minutes" is a window-based rule; "ten minutes" is the window length. There are tumbling windows (non-overlapping fixed slices, like a clock ticking), sliding windows (overlapping slices that advance continuously), and session windows (slices defined by inactivity gaps in a user’s activity). For security, the window length is usually the dial that trades detection sensitivity against state size: longer windows catch slower-burn attacks but cost more memory.

Watermarks

A watermark is the stream processor’s assertion that "I have probably seen all events with timestamps up to time T." When the watermark crosses the closing edge of a window, the processor emits that window’s result. The tricky part is that events arrive out of order. An EDR agent in a coffee shop on flaky Wi-Fi may deliver a process-execution event ten minutes after it happened. If the watermark advanced too aggressively, that event arrives after its window closed and either gets dropped or gets routed to a "late events" side channel. Tuning watermarks is the part of pipeline detection that most operators underestimate.

Stateful operators

A stateless operator examines one event in isolation. "Is this DNS query for a domain whose entropy exceeds 3.5?" is stateless. A stateful operator has to remember things across events. "Has this user logged in from a different country in the last twenty-four hours?" requires the processor to remember the last login location per user. That memory lives in a state store (usually RocksDB for Flink, an internal columnar store for RisingWave, or an external Redis or Kafka topic for some Kafka Streams deployments). State size grows with cardinality (how many users, how many hosts) and window length (how far back you have to remember), and state explosion is the most common operational failure mode in long-window pipeline detection.

Incremental view maintenance

Incremental view maintenance (IVM) is the technique of keeping a derived view (an aggregation, a join, a materialized query result) up to date by applying only the deltas from new events, rather than re-running the whole query. RisingWave is built around IVM; Materialize is the standalone-IVM database that popularized the term. For security, the practical implication is that you can write a detection as a SQL query against a continuously maintained view, and the engine handles the "keep it current as new events arrive" part automatically. That moves a lot of complexity out of the application layer and into the engine. The trade-off is that IVM engines have specific shapes of query they handle efficiently and shapes they don’t, and learning where the line is takes time.

Strong fit: stateless and short-window stateful detections

The detections that map cleanest to stream processing are the ones that can be evaluated per event or within a bounded recent window. DGA domain detection (domain entropy plus first-seen check) is stateless. Lateral movement on first-seen internal connections is short-window stateful (remember recent source-destination pairs). Impossible-travel authentication checks are short-window stateful with a per-user state slot. Credential-dumping process-execution detection is largely stateless pattern matching against process names, command lines, and parent-process context, with an optional threat-intel enrichment lookup. Ransomware behavioral detection ("more than 50 files modified with unusual extensions in 60 seconds") is a 60-second tumbling window per host.

For these shapes, pipeline detection earns its keep, because the detection fires in seconds rather than minutes, the signal stream stays small (kilobytes to low megabytes per day per rule), and the rules themselves are often more concise in stream SQL than in their SIEM correlation-language equivalents, since the engine handles windowing and state automatically.

Hard fit: long-window correlation, retroactive investigation, complex ML

Some detection patterns don’t fit stream processing well. Multi-day correlation (“account accessed from three or more countries within seven days”) creates a state explosion problem. The engine has to remember per-user activity for the full window, and at large user-cardinality this gets expensive fast. The honest pattern here is to do the short-window piece in the stream and the long-window piece as a batch query against the lakehouse, where the seven-day join is cheap because the data sits at rest in Iceberg or Delta.

Retroactive investigation (“was this account already compromised thirty days ago, and we just noticed today?”) is fundamentally incompatible with a pure pipeline model. The pipeline didn’t store the raw events thirty days ago; it stored signals and a sample. If you adopt pipeline detection without a complementary raw-data sample (typically 5–20% of events, landed in the lakehouse), you give up retroactive analysis. That trade-off is the single most common reason teams keep at least some raw-event storage even when they shift detection upstream.

Complex machine-learning detection (UEBA models with fifty-plus features computed over ninety-day windows) usually runs in the lakehouse, with the stream processor either feeding features into the model or applying a pre-trained lightweight model in-stream for fast scoring, so once ML enters the picture most of the deployments I've seen end up hybrid rather than pure-stream.

The cost case for pipeline detection is real, but the headline figures published in vendor material need to be taken apart before they’re useful. I run a 10 TB/day illustrative scenario here using public list pricing, enough to show the shape of the trade-off, not the right number for any specific deployment. Negotiated SIEM pricing, retention policy, sampling strategy, and existing infrastructure reuse all move the number.

In the query-based pattern at Splunk-like list pricing, indexed storage (10 TB/day × 90 days at $3–10/GB/month) runs roughly $2–3M/month, plus $300–500K/month for correlation and ad-hoc query compute. Total $2.5–3.5M/month; three-year TCO around $90–125M before discount.

In the pipeline pattern, stream processing costs $50K–300K/month depending on commercial vs self-managed tooling (Tenzir, Cribl, RisingWave, self-hosted Flink). Storage shrinks because you’re landing signals (typically <1% of raw) plus a 5–20% sample into S3-backed Iceberg at $0.02–0.05/GB/month rather than indexed-SIEM pricing. Total $100K–500K/month; three-year TCO around $4–18M.

That’s a 10–25× cost reduction in the illustrative case, which is the basis for the "85–95% savings" claim in vendor material. The number is plausible on this workload shape, not guaranteed, and depends on accepting that you no longer hold 100% of raw events in indexed storage. If your compliance posture requires 100% raw retention in an indexed format, pipeline detection complements rather than replaces the SIEM and the savings are smaller.

For the costing details and the cross-currents on table format choice for the storage layer underneath, see the Iceberg vs Delta Lake essay. The lakehouse choice is the larger of the two cost levers in this architecture, and pipeline detection compounds it rather than substitutes for it.

Stateless: DGA domain detection

Domain-generation algorithms produce high-entropy, long, never-seen-before domains as malware callback addresses. The detection is per-event: compute the Shannon entropy of the domain string, check the length, check whether it’s on a known-domain allowlist. No state required beyond the allowlist (which itself is a slowly changing reference table the engine joins against).

-- Stateless DGA detection as a streaming SQL view
CREATE MATERIALIZED VIEW dga_signals AS
SELECT
  event_time,
  source_ip,
  domain,
  shannon_entropy(domain) AS entropy
FROM dns_queries
WHERE shannon_entropy(domain) > 3.5
  AND length(domain) > 20
  AND domain NOT IN (SELECT domain FROM allowlist);

On RisingWave or a similar IVM engine, this materialized view stays current automatically as new DNS events arrive. Detection latency is essentially the network and engine round-trip: tens to hundreds of milliseconds.

Short-window stateful: ransomware behavior

Ransomware shows up in EDR file-modification telemetry as a burst of unusual file activity: many files modified per minute, often renamed to unusual extensions, often with high-entropy content (encrypted). The detection is a 60-second tumbling window per host, with an aggregate on file count and a check on extension-set rarity.

-- Ransomware behavioral detection: 60-second tumbling window per host
CREATE MATERIALIZED VIEW ransomware_signals AS
SELECT
  host_id,
  window_start,
  COUNT(*) AS files_modified,
  COUNT(DISTINCT file_extension) AS distinct_extensions,
  COUNT(*) FILTER (WHERE file_extension IN
    ('.encrypted', '.locked', '.crypto')) AS ransomware_exts
FROM TUMBLE(file_modifications, event_time, INTERVAL '60' SECOND)
GROUP BY host_id, window_start
HAVING COUNT(*) > 50
   AND COUNT(*) FILTER (WHERE file_extension IN
     ('.encrypted', '.locked', '.crypto')) > 0;

State requirement: per host, the running file-modification count and extension set for the current 60-second window. At 10,000 hosts that’s tens of thousands of small state entries, comfortable for any of the engines named above. Detection latency floors at the window length plus emission delay, so call it 60–65 seconds end to end. That’s slower than the DGA example, but it’s the right floor for this detection shape because the pattern itself requires observing accumulated behavior, not a single event.

Long-window stateful: impossible travel

Impossible-travel detection compares each authentication event against the last-known location for the same user, computes geographic distance and elapsed time, and checks whether the implied speed exceeds a threshold (around 1,000 km/h is the canonical "faster than commercial aviation" cut). The state requirement is one slot per user, holding last login location and timestamp. State cardinality equals the user population, which for most enterprises is bounded in the tens of thousands; again, comfortable for any of the engines. The state TTL (typically 24 hours) prevents unbounded growth on inactive users. This pattern sits at the edge of what I’d call short-window stateful; the 24-hour TTL is long by streaming standards but small by SIEM correlation standards, and that gap is roughly where pipeline detection starts to need help from the lakehouse.

The gaps in my own evidence base that I’d flag for any architect making a decision against this essay:

• Sub-100ms is the engine number, not the SOC number. Published streaming-engine latency measures the firing-to-result path inside the engine. End-to-end latency from attacker action to analyst notification still includes collector lag, network transit, signal-routing delay, and notification fan-out. The pipeline pattern shrinks the largest component (the scheduled query interval), but the residual end-to-end number may still land in the multi-second range.
• Cost numbers are illustrative, not benchmarked. The 10× cost-reduction figure is a list-price model on a 10 TB/day workload. Negotiated SIEM pricing, existing infrastructure reuse, retention-policy specifics, and sampling strategy all move the number. The shape of the trade-off is real; the specific multiple is workload-dependent.
• OCSF-shaped event streaming hasn’t been independently benchmarked. The streaming engines I name above publish benchmarks on TPC-H or synthetic workloads, not on OCSF security event shapes specifically. Performance on real EDR or CloudTrail event volumes may differ. Benchmarking against a representative production workload is a real piece of evaluation work that shouldn’t be skipped.
• Vendor-published case studies are vendor-published. The Tenzir, Cribl, and RisingWave references I’ve seen are largely Tier B (vendor blog posts, conference talks with customer logos). They’re consistent with the structural argument, but independent third-party validation at production scale (with methodology disclosed) is still relatively thin.
• Late-event handling under attack conditions is under-discussed. Watermark tuning and late-event policies get attention in normal-operations posts. What happens under DDoS conditions when collector backlogs spike, or under an active adversary intentionally delaying telemetry, is a harder operational question that I haven’t seen published cleanly.

None of these gaps invalidate the pattern, since they’re the questions a security architect should ask before committing, and the questions a vendor selling pipeline-detection tooling should be prepared to answer.