Writing · Pillar
Query engines.
ClickHouse at petabyte scale, DuckDB for analyst hunting, materialized views, push vs pull, dbt as the SQL-transformation layer.
Depth tier for ch5 (Performant) · 7 essays
Listed from the front door down to the measured evidence. Start here marks the entry essay. The reading-level tag says whether you are about to read the frame (Orientation), the argument (Depth), or the narrowest first-party finding (Evidence). The durability tag flags whether a number is durable architecture or an Evidence-pinned result tied to a version or price and meant to be re-run.
- Start here OrientationDurable
Push vs pull query engines.
Vectorized execution models for security analytics. Push engines win on throughput; pull engines win on adaptive replanning. What the difference means for your workload mix.
Read →
- DepthDurable
DuckDB for analyst-driven threat hunting.
Single-node SQL on parquet that an analyst can run from a laptop. Where DuckDB fits as the analyst-overlay tier, what it doesn't replace, and how the Lambda-native pattern works for ad-hoc hunts.
Read →
- DepthEvidence-pinned
dbt for security data.
Transformation patterns for detection engineers. The dbt-Sigma-OCSF model layer, where dbt is a clean fit, and where it competes with your pipeline tooling.
Read →
- EvidenceEvidence-pinned
One engine in front: StarRocks over shared Iceberg.
The strongest one-engine answer to the question a SOC actually asks: who is going to run four engines. StarRocks alone over the same Iceberg tables covers three of the four analyst-facing workloads from one MySQL-wire endpoint, and the scored join bench (Tier B, single host) prices the giveaways — no SOC-shaped join exceeded 1.5 s on any engine, so the spread that justified four endpoints is single-digit. The Splunk DB Connect leg stays labeled Tier-D untested.
Read →
- EvidenceEvidence-pinned
ClickHouse at petabyte scale.
Netflix's 5 PB/day ClickHouse optimization journey — fingerprinting (216 µs → 23 µs), native protocol serialization, tag sharding (3 s → 700 ms). Plus Huntress's 90%+ cost reduction migrating from Elastic.
Read →
- EvidenceEvidence-pinned
Materialized views for security data.
What actually works at petabyte scale. Where MVs are the right answer (recurring dashboards, Sigma correlations), where they fall apart (high-cardinality, schema churn), and which engines deliver the lifecycle math.
Read →
- EvidenceEvidence-pinned
Isolation-first security data platforms.
Co-tenant scheduled detection load is what breaks the interactive query experience, not raw data volume. A first-party single-host bench finds the knee at 64x the base scheduled rate and shows why a dedicated, isolated deployment is the failure you are paying to avoid.
Read →