A scoring matrix for security data tools

Read the v1 scoring writeup → — per-archetype orderings, five validation patterns, refresh triggers. 2026-05-25.

Components

• 0

  Platform Pattern — assemble the parts yourself or buy them bundled

  Composed vs. Managed (Databricks, AWS Security Lake, Snowflake)

• 1

  Lakehouse / Storage Format — the file format your data sits in on disk

  Iceberg · Delta Lake · Hudi

  → lakehouse essays

• 2

  Catalog / Metadata — the index that tracks what data exists and who can read it

  Hive Metastore · Polaris · Nessie · Unity · Glue

  → catalogs essays

• 3

  Query Engine — what runs your searches and hunts

  ClickHouse · StarRocks · Trino · DuckDB

  → engines essays

• 4

  Ingestion / Route — what collects the logs and moves them where they need to go

  Tenzir · Vector · Cribl · Kafka Connect · native shippers

  → pipelines essays

• 5

  Graph / Visualization / Analyst UX — where an analyst actually looks at the data

  Splunk SH federated · Grafana · Superset · custom · vendor SOC UIs

  → detection essays

• 6

  Storage Tier — where the bytes live and what each tier costs to keep

  S3 (Standard / IA / Glacier) · MinIO · Wasabi · NetApp · Dell ECS

  → economics essays

• 7

  Streaming Pipeline — connects ingestion to the lake and powers real-time detection

  Kafka / Confluent · Redpanda · Fluss · Flink · Spark Structured Streaming · RisingWave

  → pipelines essays

• 8

  Agentic / Authoring / Interface — how analysts and agents author, share, and work with the analytics

  Jupyter + msticpy · marimo · data-source MCP servers · LangGraph + CodeAct · SaaS copilots

  → AI essays

These nine components are the MOAR reference architecture. The matrix is the scoring view of the same vocabulary, not a second vocabulary. The 3x3 grid above scores three of them at a glance (engines, formats + catalogs, pipelines); the architecture narrative, component by component, is on the MOAR thesis page. Component 0 (Platform Pattern) is the composed-vs-managed decision that frames the other eight.

Methodology

• Each component is scored against weighted criteria specific to your environment.
• Scores: 1–5 (5 = best fit).
• Weights sum to 100, assigned per engagement based on workload.
• Final scores are weight-adjusted; ties broken by documented qualitative reasoning.
• Cross-component dependencies are explicit (catalog choice constrains query engine, etc.).

Evidence → Matrix → recommendation

The Matrix is the destination every lab benchmark rolls up into. The lab does not publish numbers for their own sake; each reproducible result becomes a 1–5 score on a component criterion, the scores are weighted by your workload archetype, and the weight-adjusted totals produce a defensible recommended bundle. The benchmarks are the evidence; the Matrix is the decision the evidence supports.

A cell-winner is not a procurement-defensible default; the recommendation is the weighted total for your archetype, not the best score on any single criterion. The worked example below shows the whole path on one component, and shows the winner change when the archetype changes.

A worked scorecard (illustrative)

One worked example of the method, end to end, on a single component, the Query Engine, for a Zeek-heavy SOC archetype (high-volume network time-series, long retention, sub-5 s p99 on hunting aggregations, a handful of concurrent analysts). The capability scores are grounded in the public lab (Tier B, single host) where the criterion is measured, and qualitative where it is client-specific. It is illustrative: the weights are an example archetype profile, not a client's, and the per-cell vendor-claim-vs-shipped-reality delta, which is published on the component pages, is left out of this illustrative table to keep it readable.

For this archetype the recommendation is ClickHouse-over-Iceberg (414), with StarRocks a close alternative (392). The gap is carried by the criteria where the lab actually separates the engines (analytical aggregation performance and compute cost) more than by any single headline. StarRocks wins the concurrency criterion outright (5 vs 4), which is the whole point of the next paragraph. The per-cell delta between each vendor's published claim and its shipped reality is public, and it rides every row above; where an engagement earns its keep is the step after this table, turning these public scores into your weighted bundle, your sequencing, and the reversibility cost of each choice against your own stack.

The winner changes with the archetype, the same cells under different weights, a mixing board rather than a leaderboard. Re-weight for a many-concurrent-analyst environment (concurrency 32, cost 15, perf 10, ops 11, routability 8, Iceberg-native 10, semantic 4, federation 4, dialect 3, skill 3) and the same cells re-total StarRocks 410 > ClickHouse 404: the ranking flips because the concurrency bench found StarRocks degrades most gracefully under load while the single-query latency edge that wins the Zeek-hunting archetype erodes to a throughput tie at the host CPU ceiling. No cell changed; the workload did. That is why the engagement scores against your archetype rather than publishing a single "best engine."

Honesty boundary: the lab evidence behind these cells is two-regime: the schema-on-read index wins the simple indexed lookups, and the lakehouse engines win the hunting-shaped aggregations, where ClickHouse reading Iceberg leads the schema-on-read baseline by ~10–11× across three independent draws (2026-06-14) while a ~4.2–4.6× open-format tax separates native MergeTree from the same engine reading Iceberg, running 5–62× on the individual hunting-shaped queries; the older 46.8× native and 10.1× Iceberg numbers are the five-query-average arm-pairings of that same result (Tier B, single host, 10M events, CV-gated, identical answers verified). The lab-backed performance cells are illustrative single-host Tier B; the TB-scale multi-node regime is unmeasured.

Augment vs replace — the decision path

The component scores answer which open stack to pick if you move; the decision path answers whether to move, and how far — the question a board votes on. The incumbent and the partial moves are scored candidate paths, not a baseline: Stay (status-quo SIEM), Augment (keep the SIEM for hot/detection, offload cold retention to the lakehouse), Hybrid-tiered, and Full-replace. The breakeven where each path pays back is computed, not asserted — migration cost divided by the monthly cost-to-serve saving against staying — so a board sees the cheapest path at its own retention horizon and the month at which the call flips. Each recommendation carries a four-part board-defensibility read: the call, what being wrong costs (the reversibility kill-switch), the evidence tier behind the crossover, and the one assumption that flips it. And the recommendation is the lowest risk-adjusted cost: a path that silently breaks a meaningful share of your detections, or fails a mandatory compliance control like WORM, is ineligible to win on cost alone. The method is here; the per-client crossover and the scored gates are the engagement.