The Capability Matrix
A scoring matrix for security data tools.
Once you require open formats, more than one engine qualifies — and every vendor's deck says theirs wins. That decision is where independent measurement earns its keep, because the measured spread is compressed at single-node SOC scale: every engine I tested answered the SOC join suite in under 1.5 s (Tier B, single host), so the pick usually turns on catalog maturity, concurrency behavior, and operational cost rather than raw speed. The Capability Matrix is the instrument for making that call: seven components, criteria weighted to your workload, scored from lab measurements rather than datasheets.
The full scored Matrix is public and stays public, weighted totals and criteria-by-criterion reasoning and the vendor-claim-versus-shipped-reality deltas included, all of it live on the component pages and refreshed quarterly, so anyone can check the evidence without paying for it. What a Security Data Works engagement (see /engagements) adds is the client-specific application: the scores are the same for everyone, but the weights are not, so your archetype weights turn those public scores into your per-workload bundle, your migration sequencing, and the reversibility cost of each choice, scored against your own stack and workload. That is how an engagement chooses between the compliant options without taking any vendor's word for it.
The scored matrix, at a glance
Three components. Three archetypes. Nine cells.
The matrix is public: every score, weight, and evidence tier below is open to read. Each cell is one component-by-archetype intersection, showing the top candidate for that intersection and colorized by how decisively it wins, and clicking any cell opens the full scoring table with weights, evidence tiers, and the discussion that follows the numbers. What you engage me for is applying this to your environment, not access to the scores.
Component 3
Components 1+2
Component 5
Evidence and audit state
Most cells currently sit at evidence Tier B–C. Five first-party benchmarks are now published, four of them with public code you can rerun, and the headline is the flagship Zeek run. The lab evidence behind these cells is two-regime: the schema-on-read index wins the simple indexed lookups, and the lakehouse engines win the hunting-shaped aggregations by 5–62× — a 46.8× native / 10.1× Iceberg average across the five-query suite (Tier B, single host, 10M events, CV-gated, identical answers verified). All cells are illustrative single-host Tier B; the TB-scale multi-node regime is unmeasured. The Tier-A upgrades are still the named Q3 2026–Q1 2027 bake-offs (catalog RBAC, multi-node OLAP federated-join, pipeline OCSF-lossiness), not claims already in hand. The lab's first external review is a Q4 2026 forward commitment; the reviewer is named on the lab page when it completes, not before.
Read the v1 scoring writeup → — per-archetype orderings, five validation patterns, refresh triggers. 2026-05-25.
Components
- 0
Platform Pattern — assemble the parts yourself or buy them bundled
Composed vs. Managed (Databricks, AWS Security Lake, Snowflake)
- 1
Lakehouse / Storage Format — the file format your data sits in on disk
Iceberg · Delta Lake · Hudi
- 2
Catalog / Metadata — the index that tracks what data exists and who can read it
Hive Metastore · Polaris · Nessie · Unity · Glue
- 3
Query Engine — what runs your searches and hunts
ClickHouse · StarRocks · Trino · DuckDB
- 4
Ingestion / Route — what collects the logs and moves them where they need to go
Tenzir · Vector · Cribl · Kafka Connect · native shippers
- 5
Graph / Visualization / Analyst UX — where an analyst actually looks at the data
Splunk SH federated · Grafana · Superset · custom · vendor SOC UIs
- 6
Storage Tier — where the bytes live and what each tier costs to keep
S3 (Standard / IA / Glacier) · MinIO · Wasabi · NetApp · Dell ECS
These seven components are the MOAR reference architecture. The matrix is the scoring view of the same vocabulary, not a second vocabulary. The 3x3 grid above scores three of them at a glance (engines, formats + catalogs, pipelines); the architecture narrative, component by component, is on the MOAR thesis page. Component 0 (Platform Pattern) is the composed-vs-managed decision that frames the other six.
Methodology
- Each component is scored against weighted criteria specific to your environment.
- Scores: 1–5 (5 = best fit).
- Weights sum to 100, assigned per engagement based on workload.
- Final scores are weight-adjusted; ties broken by documented qualitative reasoning.
- Cross-component dependencies are explicit (catalog choice constrains query engine, etc.).
Evidence → Matrix → recommendation
The Matrix is the destination every lab benchmark rolls up into. The lab does not publish numbers for their own sake; each reproducible result becomes a 1–5 score on a component criterion, the scores are weighted by your workload archetype, and the weight-adjusted totals produce a defensible recommended bundle. The benchmarks are the evidence; the Matrix is the decision the evidence supports.
How the Matrix works · evidence → recommendation
A cell-winner is not a procurement-defensible default; the recommendation is the weighted total for your archetype, not the best score on any single criterion. The worked example below shows the whole path on one component, and shows the winner change when the archetype changes.
A worked scorecard (illustrative)
One worked example of the method, end to end, on a single component, the Query Engine, for a Zeek-heavy SOC archetype (high-volume network time-series, long retention, sub-5 s p99 on hunting aggregations, a handful of concurrent analysts). The capability scores are grounded in the public lab (Tier B, single host) where the criterion is measured, and qualitative where it is client-specific. It is illustrative: the weights are an example archetype profile, not a client's, and the per-cell vendor-claim-vs-shipped-reality delta, which is published on the component pages, is left out of this illustrative table to keep it readable.
| Criterion | Wt | CH/Ice | StarRk | Trino |
|---|---|---|---|---|
| Analytical perf (hunting aggregations) | 20 | 5 | 4 | 4 |
| Cost-to-serve (compute $/effective-TB) | 18 | 5 | 4 | 3 |
| Concurrency / multi-tenant | 15 | 4 | 5 | 3 |
| Iceberg native vs connector | 12 | 3 | 4 | 5 |
| Operational simplicity | 12 | 4 | 3 | 3 |
| Routability (deterministic front end) | 8 | 4 | 4 | 4 |
| Semantic-layer / MV rewrite | 5 | 3 | 4 | 2 |
| Federation breadth | 4 | 3 | 2 | 5 |
| SPL→SQL dialect distance | 3 | 3 | 4 | 4 |
| Existing in-house skill base (client-specific) | 3 | 4 | 3 | 3 |
| Weight-adjusted total (max 500) | 100 | 414 | 392 | 358 |
For this archetype the recommendation is ClickHouse-over-Iceberg (414), with StarRocks a close alternative (392). The gap is carried by the criteria where the lab actually separates the engines (analytical aggregation performance and compute cost) more than by any single headline. StarRocks wins the concurrency criterion outright (5 vs 4), which is the whole point of the next paragraph. The per-cell delta between each vendor's published claim and its shipped reality is public, and it rides every row above; where an engagement earns its keep is the step after this table, turning these public scores into your weighted bundle, your sequencing, and the reversibility cost of each choice against your own stack.
The winner changes with the archetype, the same cells under different weights, a mixing board rather than a leaderboard. Re-weight for a many-concurrent-analyst environment (concurrency 32, cost 15, perf 10, ops 11, routability 8, Iceberg-native 10, semantic 4, federation 4, dialect 3, skill 3) and the same cells re-total StarRocks 410 > ClickHouse 404: the ranking flips because the concurrency bench found StarRocks degrades most gracefully under load while the single-query latency edge that wins the Zeek-hunting archetype erodes to a throughput tie at the host CPU ceiling. No cell changed; the workload did. That is why the engagement scores against your archetype rather than publishing a single "best engine."
Honesty boundary: the lab evidence behind these cells is two-regime: the schema-on-read index wins the simple indexed lookups, and the lakehouse engines win the hunting-shaped aggregations by 5–62× — a 46.8× native / 10.1× Iceberg average across the five-query suite (Tier B, single host, 10M events, CV-gated, identical answers verified). All cells are illustrative single-host Tier B; the TB-scale multi-node regime is unmeasured.
Augment vs replace — the decision path
The component scores answer which open stack to pick if you move; the decision path answers whether to move, and how far — the question a board votes on. The incumbent and the partial moves are scored candidate paths, not a baseline: Stay (status-quo SIEM), Augment (keep the SIEM for hot/detection, offload cold retention to the lakehouse), Hybrid-tiered, and Full-replace. The breakeven where each path pays back is computed, not asserted — migration cost divided by the monthly cost-to-serve saving against staying — so a board sees the cheapest path at its own retention horizon and the month at which the call flips. Each recommendation carries a four-part board-defensibility read: the call, what being wrong costs (the reversibility kill-switch), the evidence tier behind the crossover, and the one assumption that flips it. And the recommendation is the lowest risk-adjusted cost: a path that silently breaks a meaningful share of your detections, or fails a mandatory compliance control like WORM, is ineligible to win on cost alone. The method is here; the per-client crossover and the scored gates are the engagement.
The full scored tables
The three component matrices, in full.
Public — every score, weight, and evidence tier
Component 3 — engines · v1.3
Engines scoring matrix.
Four candidates (ClickHouse, Trino, StarRocks, DuckDB) × nine criteria × three archetypes. ClickHouse leads Archetype A on raw analytical query performance (a two-regime result: ~47× the schema-on-read SIEM baseline on native MergeTree, five-query average, 5–62× on individual hunting-shaped queries across the native and Iceberg realizations, compressing to ~10–11× over the same baseline once run on Iceberg — the ~4.2–4.6× gap between the two regimes is the open-format tax; the index still wins the simple lookups; answer-equality verified, single-node Tier B, weight 22 — not on cost, which Archetype A does not score); Trino wins B on federation breadth; Athena leads Archetype C. Weighted-final scores, weights, and evidence tiers visible per cell.
Components 1+2 — formats + catalogs · v1.3
Formats + catalogs scoring matrix.
Four paired candidates (Iceberg+Polaris, Iceberg+Nessie, Delta+Unity, Iceberg+Glue) × ten criteria × three archetypes. Iceberg+Polaris leads Archetype A; two candidates tie at Archetype B; Iceberg+Glue dominates Archetype C. 120 scored cells plus a cross-archetype synthesis table.
Component 5 — pipelines · v1.3
Pipelines scoring matrix.
Four candidates × nine criteria × two archetypes (A, B) plus a five-candidate scan at Archetype C. Includes the Tenzir-vs-Cribl procurement-defensibility split at A, where the technical winner is not the procurement-defensible default, and a 'no dominant winner' finding at C (Firehose+Lambda lacks structural lead).
Interpretation — the reasoning
Read after at least one scoring table.
Cross-cutting views, the synthesis patterns that emerge across the nine cells, the thirty-one scoring decisions, and the fifteen foundational assumptions with their refutation criteria.
Cross-cutting view
The nine cells, side by side.
Where the rankings diverge across archetypes. Why ClickHouse leads A but lags C, why Iceberg+Polaris leads A but is structurally weak at C, why the same Pipelines candidates rank differently when self-hosted versus AWS-native. Read after at least one component page.
Methodology synthesis
Five validation patterns. One non-obvious finding per archetype.
The structural reasoning that emerged from the nine cells, where the matrix becomes useful as a decision aid versus a scoring artifact. Why certain candidates differentiate, why two F+C candidates tie at Archetype B, why the procurement-defensible default at Pipelines A is not the technical winner.
Scoring decisions · 31 MDRs
The scoring decisions, summarized.
The scoring-layer decisions behind the cells, covering criterion taxonomies, archetype weights, evidence completeness rules, revalidation cadence, and disclosure discipline. The registry now holds 31 decision records (MDR-0001–0031): v1.4 adds MDR-0026 (Pipelines virtual-view criterion) plus the Moves #2/#3 decision-instrument records (MDR-0027–0030, augment-vs-replace + board read + detection-survivability + compliance/WORM, surfaced on their own pages) and MDR-0031 (superseded by the 2026-07-11 publication ruling, which dissolves the public/paid line it addressed). The full decision records are published on this page.
Foundational assumptions · 15
Fifteen foundational assumptions. Each one has a refutation criterion.
The premises beneath the MDRs. Confidence, status, basis, and the concrete observable that would refute each one. A-05 was moved to confirmed in the 2026-05-25 v1.3 cut (Unity OSS Iceberg-REST read-only finding). A-04, A-06, A-13 received first-party citations.
MethodologyThe structural layer. Audit-grade depth on how the matrix is built.
The decision framework, the seven-component criteria taxonomy, the 90-vendor evidence-tiered database, and the six tool-architecture ADRs. Most clients never need this depth on first read; it's here for the assessment-grade audit when one is requested.
Decision path · augment vs replace
Should you move, and by how much?
The scored stay-vs-go crossover (MDR-0027): the incumbent and the partial moves (Stay / Augment / Hybrid-tiered / Full-replace) as candidate paths, with the breakeven computed from the cost-to-serve cells and the cheapest path shown by retention horizon. Plus the four-part board-defensibility read (MDR-0028) and the Move #3 detection/compliance risk gate. Illustrative Zeek-heavy SOC worked example; per-client crossover is the engagement.
Detection-survivability · Component 5
How many detections survive the move?
Component 5 scored (MDR-0029): what fraction of the detection corpus survives a move, where survive means executes correctly, not compiles. The three-band model scores silent degradation worse than a clean failure, anchored to the SIGMA-EXEC evidence (PPL runs correlation windowless → silently wrong). The whole-corpus score and the correlation-subset score are reported separately so the dangerous tail can't hide. Feeds the decision-path risk gate.
Compliance / WORM · gate-capable axis
A mandated control disqualifies, it doesn't discount.
Compliance/evidentiary posture as the first gate-capable cross-cutting axis (MDR-0030): WORM, legal-hold, chain-of-custody, residency, retrieval-SLA scored 1–5, where a mandated control (17a-4 WORM) disqualifies a candidate before weighted totals rather than lowering a score. Carries the honest limit that Iceberg V3 row-lineage is blocked on the OSS pyiceberg+Nessie path. Feeds the decision-path risk gate.
Decision framework
Four phases, fourteen questions.
Sizing (S1–S4) eliminates vendors that can't carry the workload. Foundational architecture (F0–F4) sets multi-year direction. Organizational constraints (Q1–Q4) filter by feasibility. Use cases (Q5) drive the final scoring. With elimination math at each step.
Component criteria
Seven components, the scoring criteria for each.
Platform Pattern, lakehouse format, catalog, query engine, ingestion, visualization, storage tier. The criteria each is scored against, the elimination math by criterion choice, and the production evidence that anchors the scoring.
Vendor evaluations
The 90-vendor database, with evidence-tier rigor.
The vendor record format, the evidence-tier classification (A–E), three fully-worked records (ClickHouse, Databricks Lakehouse, Splunk Enterprise Security), and the vendor-claim-versus-shipped-reality patterns I track.
Tool architecture ADRs · 6
Six ADRs on how the matrix tool itself is built.
Trade-offs documented for each: scoring versus hard filtering, evidence-based versus marketing-driven ratings, progressive filtering UX, vanilla JavaScript over framework lock-in. Distinct from the scoring-decisions MDRs (matrix-output decisions).
Engagement evidence
Case studies.
The named teardowns below are public analyses of on-the-record deployments. The anonymized client case studies and the client-specific architecture decision records stay behind the client-materials gate.
Public teardown · global bank
Standard Chartered: self-managed SIEM on Databricks.
80% faster time-to-detect · 92% faster investigation · ~35% cost reduction (bank-reported, DAIS 2025).
Atlassian · Project Banyan · lakehouse-native
Atlassian's Project Banyan: security at lakehouse scale.
21B+ events queryable in under a minute · 80% lower ingest cost · OCSF on the Silver layer · Unity Catalog governance. SDW's analysis of Atlassian's public Databricks architecture; they built and run it.
Public referenceCitable and shareable.
Public-side material, listed here so nothing gets lost track of; everything here is citable and shareable, with attribution the only ask.
Reference catalog · public
The reference-architecture catalog.
Five-class catalog ordered by trust: public production teardowns, component references, vendor blueprints, methodologies, plus the production-validators wall. Free thought-leadership; the line is drawn at attribution, not at a password.
Narrative deck · public
The architect narrative deck.
The full pitch narrative: thesis, reference catalog, benchmark, fair-broker model, takeaway.
Getting the applied Matrix
The scored Matrix is already public; what an engagement delivers is the applied Matrix, your own scores re-weighted to your archetype and scored against your stack and your workload, with the recommended bundle, the migration sequencing, and the reversibility cost of each choice. The four-phase decision framework and the per-component scoring criteria are public too, published on this site. Only the anonymized case study and the client-specific architecture decision records sit behind a client-materials gate, indexed alongside the public reference catalog so nothing gets lost. To get the applied Matrix for your stack, book a scoping call.
The research page carries the hypotheses these scores rest on, and the engagements page describes how a Security Data Works engagement applies the Matrix to your own stack and workload.