Nine components. Each scored on criteria.

The pre-component decision that gates several of the choices below. A composed foundation is specialized open components chosen per layer, the default MOAR pattern, with each component evaluated independently. A managed foundation is an integrated offering where several component decisions come bundled in exchange for operational simplicity. The thesis does not prescribe one over the other; both are MOAR-compliant if they preserve open formats and engine portability at the layers that matter to the client. Scored on:

• Open-format preservation (does the pattern keep storage and table formats readable by outside engines)
• Engine portability at the layers that matter (can you swap a component without re-platforming)
• Operational simplicity bought vs. lock-in paid (the managed trade, priced honestly)
• Exit cost (what it takes to leave the managed offering once data and content live inside it)
• Cost-to-serve at retention (all-in $/effective-GB including egress, the managed-service premium, and the license model)

The foundation choice cascades: selecting a managed foundation like Databricks pre-determines Component 1 (Delta Lake) and Component 2 (Unity Catalog), so the downstream cells stop being independent. Splunk's Machine Data Lake is the managed-foundation candidate to watch, but as of early 2026 it is alpha and not yet a viable foundation. The point of scoring this first is to make the bundle explicit before the component-by-component evaluation begins.

Iceberg, Delta Lake, Hudi, or proprietary. Scored on:

• Multi-engine compatibility (Iceberg reads and writes from the widest engine set today, Delta next, Hudi a distant third; the ordering is stable across the three projects' own vendor-support pages, but SDW publishes no audited compatibility count and the earlier per-format vendor tallies came from a retired evaluation database that is no longer maintained or citable)
• Vendor-neutrality (Apache Iceberg PMC governance vs. Delta's Databricks-led governance)
• Schema evolution support and time-travel semantics
• Production scale references (Netflix at multi-petabyte on Iceberg; Apple, Airbnb on Iceberg; Walmart on Delta; Uber's mixed Iceberg/Hudi history)
• Streaming write maturity (the open engineering question; varies by quarter)
• Cost-to-serve effect (compression × open-vs-licensed cost on $/TB stored; deliberately low base weight, because a format's cost lever is mostly indirect, flowing through storage and engine)

The convergence trajectory between Iceberg and Delta, covered as a deeper analysis, softens the format-as-bet framing that was central in 2023. The catalog choice (Component 2) is now where the bet actually gets placed, not the format.

ClickHouse, Pinot, Trino, Starburst, StarRocks, Athena, Snowflake, Databricks SQL, DuckDB. This is a deliberately broader survey than the matrix's focused C3 candidate set — Pinot, Athena, Snowflake, and Databricks SQL are carried to map the full field a client might raise, not because all four reach the shortlist for a given workload. Scored on:

Latency profile (P95).

Sub-second engines: ClickHouse (a two-regime result on the published Zeek benchmark, reproducible methodology, single-node Tier B: ~47× the schema-on-read SIEM baseline on native MergeTree and ~10× on Iceberg as the five-query-average arm-pairings of the same 2026-06-14 draws, which land at ~10–11× foil-over-Iceberg on the scan-heavy hunting aggregations, the ~4.2–4.6× gap between the two arms being the open-format tax), Pinot (sub-second under high concurrency; LinkedIn's own engineering reports 100K+ QPS, a first-party operator-vendor figure, Tier C per A-15 until independently corroborated). Two-to-five-second engines: Trino, Starburst. Two-to-ten-second engines: Athena, Snowflake, Databricks SQL, where auto-scaling absorbs variance at the cost of consistent low-end latency. Half-to-three-second engines for single-user workloads: DuckDB.

Concurrency profile.

ClickHouse and Pinot handle 10–50 concurrent low-latency queries comfortably, and high-concurrency ad-hoc hunting at cluster scale (100+ analysts) leans on the distributed-coordination engines, Trino and Starburst, though that cluster regime is the gap SDW has not yet measured. What is measured sits at single-host scale, across two benches that answer different questions and should not be run together. The workload-interference bench (2026-06-14, Tier B) ran a scheduled load against an interactive probe and found the interactive p95 stays flat to a knee at roughly 32–64× the base scheduled rate, and that at that knee the failure shape differs by engine, with ClickHouse on native MergeTree the only arm classified graceful while ClickHouse-on-Iceberg and StarRocks both fail by scheduler saturation. The multi-user concurrency bench (2026-06-15, Tier B) asked the different question of how the engines behave as the analyst count rises, and there StarRocks was the best-behaved, carrying the highest QPS ceiling (6.79) and the tightest tail (p95/p50 ≈ 1.16 at N=16, against 1.30 for ClickHouse and 1.90 for Trino), which is why its concurrency cell now scores 4 at Archetypes A and B under MDR-0037. Both arms were OSS/community editions, so neither finding is claimed for the paid tiers that sell concurrency isolation. Single-user shops land on DuckDB. Mismatched concurrency profile is a frequent root cause of "the engine is fast in isolation but feels slow in production."

One caveat that's easy to conflate with the above: a materialized view or MV-like acceleration feature (a manual ClickHouse MV, for instance) makes a given query faster, but it buys noconcurrency-knee headroom on its own — the interference knee sits at the same point with or without materialization (the StarRocks-MV upper bound hit the same ~64× interference knee as base StarRocks, 2026-06-14 workload-interference bench). "Faster query" is not "helps under concurrent load," and a concurrency-sensitive architecture shouldn't expect materialized views to move this ceiling.

Operational complexity.

Self-managed ClickHouse and Pinot are high-ops; Trino is medium-ops; serverless engines (Athena, Snowflake, Databricks SQL) are very-low-ops; DuckDB is single-process and effectively zero-ops. The Q1 team-size question gates which complexity tier is operable.

Use-case fit.

Real-time SOC dashboards: low-latency engines. Ad-hoc threat hunting: high-concurrency engines. Compliance retention and batch reporting: serverless engines. Single-user laptop analysis: DuckDB. Most production stacks end up with two engines split by use case. A few end up with three. The framework handles the multi-engine case by scoring each engine for its strongest use case rather than forcing a single-engine recommendation.

Splunk SH (federated mode), Grafana, Apache Superset, vendor SOC UIs (Elastic, Sentinel, Chronicle), custom React or similar. Scored on:

• Real-time dashboard latency (sub-second P95 from query engine through the UI)
• Detection-rule expression model (vendor-specific DSL vs. SQL vs. streaming-engine native)
• Detection-survivability (MDR-0029): what fraction of the existing detection corpus survives the move, scored in three bands (survives / silently degrades / cannot port), with silent degradation scored worse than a clean failure because it runs wrong while the dashboard stays green — the scored criterion
• Multi-source query federation
• Alert lifecycle and case-management depth
• Open-source maturity (Grafana and Superset for self-hosted; Splunk SH for federation across Splunk-shaped data; vendor UIs for vendor-shaped data)
• Cost-to-serve (seat and query licensing $/yr)

The visualization choice is often more constrained by what the analysts already use than the engagement framing initially admits. The framework documents the existing analyst surface explicitly to avoid recommending a stack that requires retraining the entire SOC.

The streaming layer connects ingestion (Component 4) to lakehouse storage (Component 1) and powers real-time detection, and it splits into two tightly-interacting sub-categories. The storage backbone (Kafka, Apache Fluss as Iceberg-native streaming storage, Kafka-to-Iceberg bridges like Tableflow) and the compute (Apache Flink for stateful processing, Spark Structured Streaming when already on Spark, RisingWave for streaming SQL). Scored on:

• Store-twice vs. single-write (whether the backbone writes Iceberg natively or forces a Kafka-raw plus Iceberg-analytical duplicate)
• Iceberg-native streaming maturity (Fluss and Tableflow as the explicit-bridge candidates)
• Stateful-join cost (Flink is the most flexible and the most operationally expensive)
• Detection-content portability across the compute engine
• Operational cost of the chosen backbone + compute pair

The unresolved tension this component answers is the mismatch between Kafka's offset-based partitioning and Iceberg's analytical partitioning, which is what forces the store-twice pattern today. The most likely 2026–2028 resolution is a hybrid tiered architecture, a ClickHouse hot tier alongside an Iceberg long tier, with Fluss or Tableflow as the bridge that lets a single write serve both.