Capability Matrix · Component 3 — Engines · Scored matrix
Four engines. Three archetypes. Three rankings.
An engine answers analyst questions against the lake. Same candidates (ClickHouse, Trino, StarRocks, DuckDB) scored on the same nine criteria. The archetype shifts the weights and the workload each cell is scored against, and the ranking re-orders. ClickHouse wins Archetype A. Trino wins B. Athena joins as a fifth candidate at C and leads there, with Trino #2.
Below is the actual grid the engagement runs against: per-criterion scores 1–5, weights that sum to 100, evidence tiers (A peer-reviewed research, official standards, or named-operator production at recognized scale; B practitioner evidence and first-party lab measurements; C vendor material, bias-flagged; D speculation, never the sole citation on a scored cell), weighted totals. Every score carries citations and revalidation dates in the backing YAML files. The weights are not the only thing that moves between archetypes, because each cell is scored against that archetype's own workload rather than copied forward from the last one, so the same criterion can legitimately land on a different score, and on a different evidence tier, at A than it does at C. DuckDB's raw analytical query performance is the clearest case, scoring 2 at A where a single-node engine cannot carry 5–50 TB/day as the primary engine, 1 at B where the 10–100 TB/day federated envelope makes the primary-engine role architecturally impossible, and 5 at C where the serverless-within-AWS shapes play to what an embedded engine is actually good at; StarRocks's raw-perf cell shows the tier moving for the same reason, sitting at Tier C at A where the only evidence was a vendor multi-table claim and at Tier B at B and C where the first-party 2026-06 join bench measured the join advantage directly. These tables are a digest of the YAML criterion set; two canon criteria (cost-to-serve and routability) are scored in the engagement layer per MATRIX.md Component 3 and do not appear as columns here.
v1.4 evidence cut · 2026-07-20 · concurrency cells re-scored on the 2026-06-15 multi-user bench per MDR-0037 · revalidate by 2026-12-15 · YAML source of truth: scoring/matrix-c3-engines-A/B/C.yaml
How to read the score colors
Score colors do most of the work here. Read across a row to see how candidates spread on one criterion; read down a column to see where a candidate is strong vs. exposed. The weighted total at the bottom rolls 1–5 scores against per-archetype weights. Evidence tier under each score names what backs it. Tier A is peer-reviewed research, an official standard, or production at recognized scale with a named operator (the Pinterest and Netflix class of reference). A performance or scale claim with no named operator sits at Tier B, which also covers practitioner reports and the first-party SDW lab benchmarks. A verifiable product-spec fact also sits at Tier A under MDR-0039, narrowly: the existence, structure, or licensing of a shipped capability that a reader can check in one step at the project's own documentation or source, which is why a connector listing or an embedded-library architecture reads as Tier A while anything about how fast or how cheaply that capability performs does not. Tier C is vendor blog or marketing material, cited only with an explicit bias flag, and Tier D is speculation, which can never stand as the sole citation on a non-null score.
Archetype A — Zeek-heavy SOC
ClickHouse leads. 3.65 – 3.85 weighted.
5–50 TB/day. 90 days hot, one year warm. The workload mix is 80% recurring queries (five to ten patterns crossed with time-range filters), 15% ad-hoc threat hunting, 5% IR point lookups. Latency targets: p99 under 5 seconds on recurring, p99 under 60 seconds on ad-hoc.
Under these weights, raw analytical query performance is worth 22 points, with native IP type support carrying the other 8 of the original 30 under the v1.3 MDR-0023 split, and ClickHouse's measured lead over the schema-on-read foil carries the ranking: ~10–11× foil-over-Iceberg on the scan-heavy aggregations across three independent draws, with a ~4.2–4.6× open-format tax between native MergeTree and the same engine reading Iceberg (triple band, 2026-06-14; the older 46.8× native and 10.1× Iceberg figures are the five-query-average arm-pairings of the same result, and the index still wins the simple lookups), answer-equality verified, single-node Tier B, with the native IPv4/IPv6 types adding to the lead. Bounds reflect the one null criterion (in-house skill base, weight 5) which gets filled at engagement scoping.
| Criterion | Weight | ClickHouse | Trino | StarRocks | DuckDB |
|---|---|---|---|---|---|
| Raw analytical query performance | 22 | 5 Tier B · ~10–11× foil band | 3 Tier B · 2.67s avg latency | 3 Tier C · vendor claim | 2 Tier B · S3 I/O bound |
| Iceberg native vs connector | 15 | 4 Tier B · V3 lag | 5 Tier A · V3 first | 3 Tier C · less mature | 4 Tier B · V2 OK, V3 lag |
| Semantic layer / materialized views | 5 | 2 Tier B · manual MVs | 2 Tier B · federates MVs | 2 Tier B · async refresh | 1 Tier B · none native |
| Federation breadth | 5 | 3 Tier B · ~15 sources | 5 Tier A · 60+ connectors | 2 Tier C · MySQL-shaped | 1 Tier B · Parquet/Iceberg only |
| Operational complexity | 15 | 3 Tier B · Keeper tuning | 3 Tier B · coordinator SPOF | 3 Tier C · no managed equiv | 5 Tier A · embedded library |
| SPL dialect distance | 10 | 2 Tier B · CH extensions | 3 Tier B · ANSI portable | 3 Tier B · MySQL-compat | 3 Tier B · clean ANSI |
| Concurrency / multi-tenant | 15 | 4 Tier B · 100s OK w/ HW | 3 Tier B · SDW conc bench | 4 Tier B · SDW conc bench | 1 Tier B · ~10 concurrent |
| Native IP type support | 8 | 5 Tier B · IPv4/IPv6 native | 5 Tier B · IPADDRESS type | 2 Tier B · string only | 3 Tier B · inet ext |
| Existing in-house skill base | 5 | — engagement-scoped | — engagement-scoped | — engagement-scoped | — engagement-scoped |
| Weighted total (lower–upper) | 3.65 – 3.85 | 3.41 – 3.61 | 2.87 – 3.07 | 2.63 – 2.83 |
When ClickHouse loses to #2 or #3
ClickHouse is the call for Archetype A because raw analytical query performance is worth 22 points (native IP type carries another 8, and ClickHouse scores 5 on both) and the H3-PERFORMANCE-01 benchmark anchors the raw-perf 5 on Tier-B evidence. The one defensible exception is #2 (Trino), whose bounds (3.41–3.61) now sit just clear of ClickHouse's; #3 (StarRocks, 2.87–3.07 after the v1.4 concurrency re-score per MDR-0037) and #4 (DuckDB, 2.63–2.83) sit well back of both.
Trino holds #2 at A (3.41–3.61) on the published table, lifted there by the v1.3 native IP type criterion (IPADDRESS is first-class) on top of its federation breadth (5) and Iceberg V3 currency (5). It still isn't the primary-engine pick under Archetype A's raw-perf weighting, but it is the natural secondary federation tier for a multi-year SIEM migration: keep Splunk in place, federate to Iceberg via Trino, drain volume to ClickHouse over time.
Where the Archetype-A ranking is fragile
The OLAP bake-off has started landing: the single-host engine-join slice ran 2026-06-10 and the multi-user concurrency slice ran 2026-06-15, while the cluster, federated-join, and AWS-native scenarios (per tests-benchmarks-plan.md) remain open. Two specific places where the remaining slices could move this ranking:
- Trino without federation. Score 3 reflects Trino's measured 2.67s average on the Zeek workload, which the 2026-05-25 first-party federation smoke re-attributes as roughly 91% base-engine cost and only ~35.6 ms (~9%) actual federation overhead at toy scale, so the old "federation floor" label overstated the federation share. The 2026-06 engine-join bench (Tier B, single host, 10M–60M-row tables over shared Iceberg) gives a first single-source read: with no federation in play Trino answered every SOC join in under 1.5s, so the floor does drop, but it also carried the field's worst join overhead at 4.35× its own flat-scan time against roughly 1.8× for StarRocks and both ClickHouse arms. The v1.3 native IP type criterion has already lifted Trino to #2 at A (3.41–3.61); a raw-perf climb from 3 to 4 on the SOC shapes would carry another 22 weighted points and put Trino at 3.63–3.83, which lands back inside ClickHouse's 3.65–3.85 band and would undo most of what the v1.4 concurrency re-score cost it, while the join-heavy shapes argue against that climb, so the rescore cuts both ways.
- StarRocks independent benchmark — now run. The 5–30s multi-table claim vs. ClickHouse 30–120s was Tier-C vendor evidence; the 2026-06 engine-join bench (Tier B, single host, 10M–60M-row tables over shared Iceberg) is the independent test it asked for, run on TPC-H-derived and SOC-shaped joins rather than the Zeek workload, and it splits the claim in two. StarRocks 4.1 measurably leads joins, winning four of the five TPC-H-derived join queries and running the heaviest six-table join 2.7–2.8× faster than both ClickHouse arms, but the spread stays small at this scale because the SOC join suite never exceeded 1.5s on any engine, and the TB-scale 5–30s vs. 30–120s framing is a different regime the bench neither confirms nor refutes. Moving raw-perf from 3 to 4 on this first-party evidence would lift StarRocks's band to 3.09–3.29 on top of the v1.4 concurrency re-score (2.87–3.07) — clear of DuckDB (2.63–2.83) though still behind Trino's 3.41–3.61 — and defensible for ad-hoc-heavy and join-heavy workloads.
Archetype B — multi-source federated lakehouse
Trino leads. 3.90 weighted. ClickHouse at #2.
10–100 TB/day. Workload mix: 50% recurring (security analytics dashboards, alerts), 30% federated joins across CMDB + EDR + identity + Zeek, 20% ad-hoc and ML feature extraction. Latency targets: dashboards p99 under 10s; federated joins p99 under 30s. 1-year hot Iceberg, 7-year cold.
The archetype shift is the most dramatic in the matrix so far. The same four candidates as A, scored on the same nine criteria against a different workload and under different weights, produce a different ranking. Federation breadth jumps from weight 5 to weight 20. Semantic-layer support jumps from 5 to 15. Iceberg-native rises from 15 to 25. Raw query perf drops from 22 to 15 and native IP type from 8 to 5. Concurrency drops from 15 to 5. In-house skill base drops to 0 ("assume hire" per MDR-0017), which is why the weighted totals collapse to single values with no bounds spread.
| Criterion | Weight | Trino | ClickHouse | StarRocks | DuckDB |
|---|---|---|---|---|---|
| Raw analytical query performance | 15 | 3 Tier B · 2.67s avg latency | 5 Tier B · ~10–11× foil band | 4 Tier B · SDW join bench | 1 Tier B · single-node ceiling |
| Iceberg native vs connector | 25 | 5 Tier A · V3 first | 4 Tier B · V3 lag | 3 Tier C · less mature | 4 Tier B · V2 OK, V3 lag |
| Semantic layer / materialized views | 15 | 2 Tier B · federates MVs | 2 Tier B · manual MVs | 2 Tier B · async refresh | 1 Tier B · none native |
| Federation breadth | 20 | 5 Tier A · 60+ connectors | 3 Tier B · ~15 sources | 2 Tier C · MySQL-shaped | 1 Tier B · Parquet/Iceberg only |
| Operational complexity | 10 | 3 Tier B · FTE overhead | 3 Tier B · Keeper tuning | 3 Tier C · no managed equiv | 5 Tier A · embedded library |
| SPL dialect distance | 5 | 3 Tier B · ANSI portable | 2 Tier B · CH extensions | 3 Tier B · MySQL-compat | 3 Tier B · clean ANSI |
| Concurrency / multi-tenant | 5 | 4 Tier A · Pinterest 17k+ nodes | 4 Tier B · 100s OK w/ HW | 4 Tier B · SDW conc bench | 1 Tier B · ~10 concurrent |
| Native IP type support | 5 | 5 Tier B · IPADDRESS type | 5 Tier B · IPv4/IPv6 native | 2 Tier B · string only | 3 Tier B · inet ext |
| Existing in-house skill base | 0 | — weight 0 · assume hire | — weight 0 · assume hire | — weight 0 · assume hire | — weight 0 · assume hire |
| Weighted total | 3.90 | 3.50 | 2.80 | 2.35 |
When Trino loses to #2 or #3
Trino takes Archetype B because federation breadth jumps to weight 20 and Iceberg-native to weight 25, and Trino scores 5 on both, so the 60+ connectors reach the CMDB + EDR + identity + Zeek join surface the archetype's 30% federated-join mix runs against while the V3-first Iceberg currency lands the recurring dashboards on native tables. Pinterest's 17k+ node Monarch production validates Trino + Gravitino + Iceberg at the scale Archetype B contemplates, so the 3.90 weighted total rests on measured ground. But two scenarios flip the call.
ClickHouse replaces Trino when the workload has under 20% federated joins and dashboard pre-materialization is acceptable ops overhead. ClickHouse's raw-perf advantage carries the recurring-heavy shapes despite the federation gap, because the 5 on raw-perf at weight 15 against Trino's 3 recovers much of what Trino banks on federation breadth and Iceberg currency, though it doesn't fully close the 0.40 spread, which is why ClickHouse holds at #2 (3.50) rather than taking B outright.
StarRocks at #3 (2.80) isn't a standalone challenger at B. It lands the join-shaped queries well, but the federation and Iceberg-native weights that dominate this archetype leave it behind, so its place here is the dual-engine pattern (StarRocks for the TPC-H-shaped joins, ClickHouse or Trino for the rest) rather than a primary-engine pick.
Where the Archetype-B ranking is fragile
- ClickHouse's raw-perf edge at weight 15. The criterion most likely to pressure the ranking. Trino wins B on federation breadth and Iceberg currency, but if the real workload turns out heavier than 20% on raw single-source scans, ClickHouse's 5 on raw-perf against Trino's 3 narrows the 0.40 gap quickly, and under-20%-federation shapes are exactly where ClickHouse takes B outright.
- Trino's raw-perf measurement at score 3. The 2.67s federation latency floor was measured under federation. Single-source Trino (Iceberg + workers, no fed planner) could drop that floor below 1s and lift the score to 4, which would only widen Trino's lead at B, so this fragility runs in Trino's favor rather than against it.
- Federation depth is the criterion that carries B. Trino's 3.90 rests on the score 5 at federation breadth (weight 20) plus the score 5 at Iceberg-native (weight 25). If an engagement's sources collapse to two or three well-supported classes, that breadth advantage stops paying, and a raw-perf-first engine closes the distance on the recurring half of the mix.
Archetype C — serverless-within-AWS
Athena leads. 4.29 weighted. Trino at #2, 3.98.
AWS-committed team. Iceberg + Glue assumed as the catalog + format pair (per the Component 1 + 2 Archetype C selection). Engine selected for Athena-shaped AWS-managed-service fit. Five candidates: the four from A and B plus AWS Athena, promoted from implicit reference to explicit candidate at v1.2.
Archetype C's weights tilt toward Iceberg-native (25), ops simplicity (20), and federation (15). Raw query perf drops to 11, with the v1.3 native IP type criterion at 4. The criterion set under MDR-0021 shares only three names with A/B (raw perf, native IP type, Iceberg-native); the other six are renamed for the serverless framing (ops_simplicity, multi_engine_federation, concurrency_at_load, splunk_spl_compatibility_or_migration_aid, semantic_layer_capabilities, in_house_skill_base), but the underlying dimensions are the same. Weights sum to 100; weighted totals collapse to single values because in_house_skill_base now gets a score (Athena 4, Trino 3, etc.) rather than holding null.
| Criterion | Weight | Athena | Trino | ClickHouse | DuckDB | StarRocks |
|---|---|---|---|---|---|---|
| Iceberg native vs connector | 25 | 5 Tier A · Glue native | 4 Tier A · mature connector | 4 Tier B · read-strong | 3 Tier B · ext less mature | 3 Tier B · less validated |
| Ops simplicity | 20 | 5 Tier A · pure serverless | 5 Tier A · EMR / Athena | 3 Tier B · oversells turnkey | 4 Tier A · Lambda-native | 2 Tier B · Cloud emerging |
| Multi-engine federation | 15 | 5 Tier A · 30+ connectors | 5 Tier A · original | 3 Tier B · REST + dict | 2 Tier B · S3+PG only | 3 Tier B · ext catalogs |
| Raw analytical query performance | 11 | 3 Tier B · 5–30s cold | 3 Tier B · fed tax | 5 Tier B · ~10–11× foil band | 5 Tier B · in-memory fast | 4 Tier B · TPC-H joins |
| Concurrency at load | 10 | 3 Tier B · 20 wg cap | 3 Tier B · wg overhead | 4 Tier B · shard config | 1 Tier B · ~10 concurrent | 4 Tier B · FE/BE split |
| Splunk / SPL migration aid | 5 | 4 Tier B · fed query connector | 3 Tier B · fed connector | 4 Tier B · positioned target | 2 Tier C · no SPL bridge | 2 Tier C · no SPL bridge |
| Semantic layer capabilities | 5 | 2 Tier B · views/MVs | 2 Tier B · views/MVs | 2 Tier B · manual MVs | 1 Tier B · views only | 2 Tier B · MVs |
| Native IP type support | 4 | 4 Tier B · Trino lineage | 5 Tier B · IPADDRESS type | 5 Tier B · IPv4/IPv6 native | 3 Tier B · inet ext | 2 Tier B · string only |
| In-house skill base | 5 | 4 Tier B · lowest ramp | 3 Tier C · EMR config | 3 Tier C · CH knowledge | 3 Tier C · low ramp | 2 Tier C · less common |
| Weighted total | 4.29 | 3.98 | 3.65 | 2.92 | 2.82 |
When Athena loses to #2 or #3
Athena leads Archetype C by 0.31 over Trino (3.98), a clear margin rather than the noise-floor tie the previous candidate set produced. Athena wins on the two criteria this archetype weights hardest, Iceberg-native (5, Glue-native) at weight 25 and ops simplicity (5, pure serverless) at weight 20, and it holds the top skill-base score because standard SQL on a managed service has the lowest ramp. The question at C is which managed path fits, not whether Athena is the default.
Trino at #2 (3.98) is the challenger when the workload genuinely spans clouds (Snowflake, Postgres, GCP BigQuery via connectors) and Athena's AWS-centric federation isn't enough. Trino matches Athena on multi-engine federation (both 5) and sits one step back on Iceberg-native (4 vs. 5). Trino-on-EMR is the managed path that earns the ops-simplicity 5; self-managed Trino-on-EKS would drop to 3 there and pull the weighted total to ~3.58, which is what keeps the 0.31 gap to Athena from closing on its own.
ClickHouse at #3 (3.65) is the pick when raw single-source query speed outweighs the AWS-managed-serverless premium, since ClickHouse carries the raw-perf 5 (~10–11× the foil over Iceberg, triple-draw; 46.8× is the native five-query-average arm-pairing) and a native IP type 5 where Athena sits at 3 and 4. What holds it at #3 is ops simplicity: the score 3 reflects A-11-clickhouse-cloud-oversells-turnkey, so ClickHouse only challenges at C where a team is willing to run the engine rather than lean on pure serverless.
Where the Archetype-C ranking is fragile
- ClickHouse Cloud's AWS-managed maturity. Score 3 on ops simplicity reflects
A-11-clickhouse-cloud-oversells-turnkey. If ClickHouse Cloud ships an Athena-compatible REST endpoint and matches Athena's true-serverless story, ops lifts to 5 and ClickHouse's weighted total climbs to ~4.05, tying or beating Trino. - DuckDB-on-Lambda concurrency ceiling. Score 1 on concurrency at load is driven by
A-14-duckdb-10-concurrent-ceiling. If MotherDuck or a DuckDB multi-host pattern ships at scale, score could lift to 3 and DuckDB's total to ~3.12, putting it more decisively ahead of StarRocks.
Cross-archetype view
Every candidate has its archetype shape.
The matrix's most important methodology validation: same four candidates, same criterion set, same scoring rubric, three different rankings. The archetype determines the ranking, not the engine's marketing or my prior. Below is the consolidated view.
| Engine | A (Zeek SOC) | B (federated) | C (serverless-within-AWS) | Home archetype |
|---|---|---|---|---|
| ClickHouse | 3.65–3.85 · #1 | 3.50 · #2 | 3.65 · #3 | A — Zeek-heavy single-source recurring |
| Athena | not scored | not scored | 4.29 · #1 | C only — AWS-managed-serverless |
| Trino | 3.41–3.61 · #2 | 3.90 · #1 | 3.98 · #2 | B — federated lakehouse |
| StarRocks | 2.87–3.07 · #3 | 2.80 · #3 | 2.82 · #5 | None — dual-engine pattern with ClickHouse |
| DuckDB | 2.63–2.83 · #4 | 2.35 · #4 | 2.92 · #4 | C-Lambda pattern — analyst overlay only |
ClickHouse peaks at A and declines by rank through B and C. Its strength is raw single-source recurring query performance, and that weight halves as the archetype shifts toward federation and AWS-native ops, so it moves from #1 at A to #2 at B to #3 at C. Trino runs the near-inverse: it lifts monotonically A → B → C (3.41–3.61 to 3.90 to 3.98) as federation breadth becomes design-center, taking #1 at B outright once federation and Iceberg-native carry the weight, then holding #2 at C behind Athena's managed-serverless fit while the v1.3 native IP type criterion keeps it at #2 at A as well.
The two candidates without a home archetype tell their own story. StarRocks's right home is the dual-engine pattern (ClickHouse + StarRocks split by query shape), and the 2026-06 join bench plus the earlier 100M-row single-table matrix (both Tier B, single host) measured exactly that split: StarRocks won four of five TPC-H-derived joins while ClickHouse took the most aggregation-shaped one, and StarRocks won no single-table shape at all on the 100M matrix. The 2026-06-15 concurrency bench repeats the pattern under load, where StarRocks's MPP scheduler carried the tightest tail of the measured arms. The specialization is real and small at this scale, which is the case for splitting by query shape rather than crowning either engine. Scored as a standalone primary engine under any archetype, StarRocks underperforms. DuckDB's right home is the analyst-overlay tier on top of a lakehouse: the Okta Lambda-DuckDB pattern, or the DNB marimo + Ibis + DuckDB on Snowflake stack. Forced into the primary-engine role, it fails on concurrency and single-node throughput.
Cross-component coupling
Engine choice constrains four other components.
The engine ranking above isn't standalone. Each candidate's score carries explicit dependencies on the format (C1), catalog (C2), pipeline (C4), and storage tier (C6). Picking ClickHouse forecloses some C2 paths; picking Trino changes the C6 cold-tier story; the DuckDB analyst-overlay pattern reshapes the C4 routing design. Here are the cross-component constraints that ride this ranking.
C1 (table format)
Iceberg is the assumed format across all three archetypes. Engines differ on V3 currency: Trino ships V3 features first, ClickHouse and DuckDB lag, StarRocks lags most. If the engagement requires V3 row-lineage for compliance erasure today, Trino's Iceberg-native score of 5 becomes what the decision rests on, particularly under the v1.3 evidence cut where the Spark + V3 reachability path splits into Spark + Hadoop (works, row-lineage confirmed) vs. pyiceberg + Nessie (does not).
C2 (catalog)
Polaris and Nessie work with all four candidates. AWS Glue is the Archetype-C default and Athena's only catalog. Unity Catalog requires UniForm or external-table indirection for ClickHouse, Trino, DuckDB, and StarRocks alike. If the engagement is Databricks-led (meaning Unity is mandatory), that indirection cost lands on every OLAP candidate, so the catalog choice, not the engine ranking, becomes the binding constraint at every archetype where Unity is in scope.
C4 (pipeline / ingest)
For any engine leaning on materialized views or an MV-like semantic layer to hit the recurring latency targets, ingest must arrive OCSF-normalized, because schema drift breaks MV matching. For the DuckDB analyst-overlay pattern, routing must be topology-aware (hot recurring queries route to ClickHouse or Trino, hunting investigations route to DuckDB). For ClickHouse's manual MV strategy, schema discipline upstream is non-negotiable. The dependency runs the other direction too: OCSF normalization at-source is scored on two levels in order, because the first binds before the second matters. Availability comes first (does the tool ship a usable mapping for the source at all, in the shape the client's actual shipper emits), and first-party pipeline-fidelity work (Tier B, single host, OCSF 1.8.0, version-bound; sdw-lab-benchmarks/pipeline-normalization-fidelity/FINDINGS-2026-06-14.md) found shipped-mapping availability, not per-field fidelity, to be the binding constraint across the open tools tested, with several common security sources having no usable shipped mapping at all, or one bound to a raw input shape (EVTX/XML) that pre-parsed EDR JSON doesn't match. Fidelity comes second, and the gap there is that a shipped mapping can get the OCSF class right and most field values right while still getting the activity classification wrong: "maps to OCSF" is a coverage claim, not a fidelity guarantee, and the part that fails is often the part detection content keys on.
C6 (storage tier)
Cold-tier story varies by engine. ClickHouse cold-tier on object store works but is less proven than Trino-over-Iceberg-on-Glacier. If the engagement has a hard cold-tier query requirement (compliance retention beyond 1 year with queryable archive), that pulls Trino up half a point on weighted total at the archetypes where cold-tier is binding. DuckDB queries S3 via pushdown. At Archetype B's 10–100 TB/day ingest, egress costs and region co-location matter, and Archetype C's serverless-within-AWS Athena pattern sidesteps the cross-region question entirely.
v1.3 evidence cut · 2026-05-25
A-06 Spark + V3 reachability split
Assumption A-06 (Iceberg V3 engine support is rolling) was refined in the v1.3 evidence cut. Two paths to Spark + Iceberg V3 row-lineage diverge in production:
- Spark + Hadoop catalog: row-lineage confirmed reachable; the path validated for Trino's Tier-A score 5 on Iceberg-native.
- pyiceberg + Nessie catalog: row-lineage does not propagate end-to-end; the path is blocked.
Practical engagement note: if a customer is on Nessie + pyiceberg and needs V3 row-lineage for compliance erasure, the catalog choice or ingest framework needs to change before the engine ranking applies. The engine's catalog support helps at the read side, but the pyiceberg-write client is the limiting layer, not the engine.
What I did not score
- Apache Spark SQL. Reserved for lakehouse-style scoring; the H3-PERFORMANCE-01 benchmark does not cover it. Needs separate evidence collection before a future scored run.
- Apache DataFusion. Embedded in the Cloudflare-ClickHouse-DataFusion production teardown, but not yet a standalone candidate. Defer to v2.
- Snowflake and BigQuery at Archetype C. Cross-cloud candidates fall outside Archetype C's "AWS-committed" premise. Relevant to a separate non-AWS archetype that has not yet been authored.
- Databricks SQL / Lakewatch. Lakewatch is a managed agentic SIEM on top of Delta + Unity, not a standalone engine choice for Iceberg + Glue. Out-of-scope for Archetype C; relevant to a Delta-format archetype.
- Engagement-specific in-house skill base. Held null at the standing-matrix level for A; weight 0 ("assume hire") at B. Filled per engagement at scoping.
Three components scored. The matrix.
Engines is one of three primary scored grids. Formats + catalogs (Component 1 + 2) is the upstream constraint; pipelines (Component 4) is the downstream ingest design that determines whether materialized views match analyst queries effectively.
Primary scored grids
Enrichment
- Synthesis · across-component patterns →
- Scoring decisions · how scores are anchored →
- Assumptions · A-series tracking →
Methodology