Writing
Essays from a practitioner
I wrote these essays to work through the decisions you face building a security data platform from open parts, from the table format and catalog up through detection, migration and cost. They're the reasoning behind the Capability Matrix scores, and the head-to-head benchmark evidence they cite lives in the Lab.
If you're new here, start with one of the reading paths below, three essays each on a single question. If you already know which decision is in front of you, browse the full catalog by pillar, where each pillar marks its own start-here essay and the pillars run as the dependency stack: the foundations (lakehouse formats, catalogs) up through OCSF and Sigma standards, engines and pipelines that sit on top of them, the detection and migration practices that consume them, and finally the economics and the AI pillar (securing the AI you deploy, and tracking where it's heading) that frame the whole. The architecture itself, component by component, is the MOAR Stack, where each piece carries its entry essay, depth, and the evidence underneath it.
Reading paths
If you read three, read these.
The full essay collection is a lot to land on cold. Each path is a three-essay arc through one question, cross-pillar, in order.
Leaving Splunk without breaking detections
The cost case, the migration trap most teams walk into, and what the timeline actually costs.
Whether you can trust your data
The quietest failures in security data, in the parsing layer and in the measurement problem underneath it.
Picking the query engine
Where each engine wins, from petabyte-scale detection down to an analyst's laptop.
When hunting becomes data science
The path threat hunters are already on, made reproducible.
Securing the AI you're deploying
One agent risk told three ways, the AI attack techniques the defensive ontology can't yet reach, and why a more capable model makes a wrong answer over security data quieter rather than rarer.
- 1The lethal trifecta, explained three ways
- 2The blind quadrant over AI systems
- 3The better the model, the quieter the wrong answer
Evidence: The ATLAS Wall: which AI controls to deploy first →
Whether your detections are telling the truth
The silent mapping failure that lets a rule compile clean and never fire, the gap between a field's shape and its meaning, and what no single tool can see about your environment.
Picking the table format
What the open stack is made of, the format decision on its production record, and the measured finding that how data enters decides more than which format holds it.
Browse by pillar
The full catalog, one pillar at a time
12 essays
Lakehouse foundations.
Open table formats and the interop layer beneath them. Iceberg, Delta, V3/V4 features, and the Arrow standards that make engine portability real.
Open the Lakehouse foundations pillar in depth order → · Depth tier for ch5 (Performant) and ch2 (MOAR assembled)
- Start here OrientationEvidence-pinned
Iceberg vs Delta Lake for security data.
Production evidence from Netflix (which built Iceberg and runs it at petabyte scale), Insider (a 90% S3 request-cost reduction), Adobe (5,000+ Delta tables), and InMobi (GDPR/CCPA on Delta) anchors a format decision that governs engine portability, vendor neutrality, and migration cost.
Read →
- DepthDurable
MOAR, explained.
What replaces and upgrades the ELK stack for security, component by component: Iceberg for the table, Arrow for the wire, D3FEND for the ontology, OCSF for the schema, Sigma for the detections, and a query engine you can change your mind about. Modular Open Architecture explained, and why being able to swap any one piece without rebuilding the rest is what makes it safe to adopt.
Read →
- DepthEvidence-pinned
Iceberg V3 changed the security lakehouse thesis.
Earlier essays argued Iceberg over Delta on vendor-neutrality. V3's Puffin deletion vectors, Variant type, row-level lineage, and default values plus V4 proposals change the trade-off space. Some prior recommendations need revision; others are reinforced.
Read →
- DepthEvidence-pinned
V4 relative paths vs DuckLake's database-metadata.
The two address different facets of the same problem space. What V4's relative paths solve, where DuckLake's database-as-metadata wins, and which security-data workloads each fits.
Read →
- DepthEvidence-pinned
Iceberg table maintenance at scale.
Compaction, snapshot expiration, orphan file cleanup. What goes wrong at petabyte scale, what the maintenance budget looks like, and where vendor catalogs help or hurt.
Read →
- DepthEvidence-pinned
Deletion vectors and GDPR.
Iceberg's right-to-erasure story: what Puffin deletion vectors actually deliver, where the gaps remain, and what an audit looks like.
Read →
- DepthEvidence-pinned
Variant fixes the CloudTrail flattening bug, within limits.
The Iceberg V3 Variant type is the structural fix for what flattening anti-patterns broke: store CloudTrail's nested JSON, EDR alerts, anything semi-structured natively and query it natively, with the engine-support and round-trip limits that still decide whether it travels.
Read →
- DepthEvidence-pinned
Row lineage as the missing CDC primitive.
Iceberg V3 row lineage may close the audit-trail gap that has dogged lakehouse detection engineering. What works in Spark 4.1 today (experimental), why Trino, ClickHouse, and DuckDB support still gates production use, and what an end-to-end CDC pattern looks like.
Read →
- DepthEvidence-pinned
Arrow and ADBC: a foundational pillar.
Why Arrow Database Connectivity is the wire-protocol layer that lets you swap engines without rewriting client code. The columnar-throughput case versus JDBC/ODBC.
Read →
- DepthEvidence-pinned
Arrow Flight and Flight SQL.
The columnar wire protocol for security data. Where Flight SQL fits in a federated stack, what it costs at the analyst's tool layer, and which engines actually speak it.
Read →
- EvidenceEvidence-pinned
The write pattern is the architectural decision.
Iceberg and DuckLake are nearly read-neutral on identical data but diverge sharply on how data enters. Over a 10-to-200-file ladder Iceberg's planning grows 17.6x while DuckLake's SQL-catalog stays flat; on tiny streaming commits DuckLake's inlining runs 2-4x faster with zero data files. Map the format to the write contract: streaming hot tier versus bulk and forensic cold tier.
Read →
- EvidenceEvidence-pinned
The encoder is the read lever, not the table format.
Register byte-identical Parquet into both an Iceberg and a DuckLake catalog, read both with DuckDB, and at a billion rows three of four queries come back at parity. The read-speed difference people attribute to the format is the Parquet writer: at a matched codec PyArrow writes 193 MB where DuckDB writes 114 MB on the same data. So pick the format on the write path, since a read benchmark mostly measures the writer.
Read →
4 essays
Catalogs.
Polaris, Unity, Nessie. Governance reach, RBAC depth, meta-catalogs for asset context, and the lock-in surface where most lakehouse buyers underestimate the risk.
Open the Catalogs pillar in depth order → · Depth tier for ch5 (Performant)
- Start here OrientationEvidence-pinned
The catalog became the control plane.
The 2026 consensus is that the catalog is the new control plane and the AI grounding layer. Right for analytics, an open question for security: enforcement is distributed, delegated, beta, and non-portable. With first-party lab evidence on the half nobody scores.
Read →
- DepthDurable
Catalog governance without native support.
When the catalog you picked doesn't carry the governance reach you need. Compensating patterns, what they cost, and which gaps don't close cheaply.
Read →
- DepthEvidence-pinned
Meta-catalogs and asset context in federated environments.
When a single catalog can't span the environment, the meta-catalog layer becomes essential. Patterns, the asset-context join problem, and what production deployments actually do.
Read →
- EvidenceEvidence-pinned
Unity Catalog vs Polaris vs Nessie.
Choosing the catalog for security data. RBAC depth, multi-engine reach, branching semantics, the procurement-defensibility math. Built on the public documentation, with the behavior I couldn't test labeled as documented rather than measured.
Read →
12 essays
OCSF & schema.
Normalization, mapping, and the anti-patterns from migrations gone sideways. Schema-on-read vs schema-on-write, OCSF reverse mapping, flattening detection logic.
Open the OCSF & schema pillar in depth order → · Depth tier for ch4 (Well-connected)
- Start here OrientationEvidence-pinned
Schema-on-read vs schema-on-write.
Splunk at $31K/month for 1 TB/day. Elasticsearch with ECS at $8–12K. Hybrid lakehouse (raw on cheap object storage, OCSF on warm) at $7.5K with parity on detection-engineer workflows. The schema-on-read tax compounds at retention scale.
Read →
- DepthEvidence-pinned
What two practitioners told me at RSA.
A field report from two RSA conversations, where practitioners described both halves of the open architecture already running in production: petabyte-scale OCSF normalization on one side, columnar storage-and-query economics on the other. The mapping labor is the real cost, and every number is attributed and discounted, because two conversations make a field report and can't do a benchmark's job.
Read →
- DepthDurable
OCSF ontological grounding: D3FEND for federal-ready.
OCSF anchored to MITRE D3FEND gives the ontology you need for federal-grade detection. What works today, what's still aspirational, and where the gaps sit.
Read →
- DepthDurable
How much ontology does security data actually need?
Security keeps being told its telemetry needs a formal ontology underneath it. Once you measure it, D3FEND grounding is reciprocal at the class level but roof-only beneath, and the architecture that would make a richer one pay off sits beside the data as a virtual layer instead of embedded in it. Use the formal model at design time as a validation oracle, the columnar schema at runtime. Keep McComb's diagnosis, drop the triplestore prescription; AI maps, humans model.
Read →
- DepthEvidence-pinned
OCSF reverse mapping.
Answering the legal-team objection. When OCSF normalization erases the original-event fidelity required for chain-of-custody, here's how reverse-mapping preserves the evidentiary record.
Read →
- DepthEvidence-pinned
OCSF and operational technology.
OCSF has no native fields for Modbus, DNP3, BACnet, or S7comm. The Issue #1515 proposal adds six fields under an ics namespace, anchored on production-validated Zeek-to-OCSF mapping work. The architecture argument for IT/OT convergence at the schema layer.
Read →
- DepthDurable
The field-mapping anti-pattern.
Field-by-field mapping during SIEM-to-lakehouse migration looks like the safe play and drops detection coverage on the way. Five patterns that break and what to do instead.
Read →
- DepthDurable
Flattening away your detection logic.
Migrating from SIEM to lakehouse is semantic translation. Treating it as schema conversion is how detection coverage breaks: flattening CloudTrail's nested JSON silently broke a privilege-escalation detection for six weeks at a financial services firm.
Read →
- EvidenceEvidence-pinned
LLM-assisted OCSF mapping.
What the migration tax actually looks like when you use LLMs to translate vendor schemas to OCSF. Where it accelerates, where it produces silent-loss errors, and how to validate.
Read →
- EvidenceEvidence-pinned
Six schemas into OCSF: the mapping is the hard part.
Field-level crosswalks of Splunk CIM, Google Chronicle UDM, Microsoft Sentinel ASIM, Elastic ECS, OpenTelemetry, and Zeek into OCSF 1.8.0. The empty cells are the finding: five recurring gaps, most of them the standard's own (missing disposition, missing certificate class, an invented-severity contract).
Read →
- EvidenceEvidence-pinned
From field mappings to the controls layer.
Up one layer from the schema crosswalks: OCSF class to digital artifact to D3FEND defense to ATT&CK offense to NIST 800-53 / SCF control. Measured hop by hop (79 D3FEND techniques to 402 controls via 606 SKOS edges; 98% of the defensive matrix reaching a governance control through ATT&CK), including the one direct link that does not exist.
Read →
- EvidenceEvidence-pinned
The blind quadrant over AI systems.
MITRE's own D3FEND dashboard scored ATLAS at 0.0% coverage over AI systems, one of three frameworks at zero. Working the mapping under an exhaust-existing-vocabulary discipline (the BFO placement rule that keeps model, event, and defensive-method apart) moves 117 of 140 techniques into the matrix, acceptance-tested 0 to 117 in MITRE's own container with every other framework byte-identical. Only five techniques genuinely need a new artifact noun, and three independent communities (D3FEND, the BOM standards, OCSF) turn out to have found the same missing vocabulary.
Read →
3 essays
Sigma & detection portability.
Sigma 2.0 correlations, pySigma backend reality, and Sigma as the fourth foundational standard alongside Iceberg, Arrow, and OCSF.
Open the Sigma & detection portability pillar in depth order → · Depth tier for ch4 (Well-connected), Sigma as the fourth standard
- Start here OrientationDurable
Sigma and detection portability.
The fourth foundational standard. Why Sigma sits alongside Iceberg, Arrow, and OCSF as the standards that decouple security data from any single vendor's analytics engine.
Read →
- DepthDurable
Why Sigma won the detection-sharing decade.
A decade of attempts to share security use cases (Sysmon configs, hunt notebooks, MITRE CAR, Atomic Red Team, Sigma), and only some endured. Quality didn't decide which ones lasted, since the best Sysmon config froze in 2021. Five structural properties predict what lasts, and Sigma has them by construction.
Read →
- EvidenceEvidence-pinned
Sigma 2.0 correlations and the pySigma backend reality.
Sigma 2.0 added correlation semantics that the backends haven't fully caught up to. Which backends ship the new constructs, which don't, and where that breaks in production.
Read →
7 essays
Query engines.
ClickHouse at petabyte scale, DuckDB for analyst hunting, materialized views, push vs pull, dbt as the SQL-transformation layer.
Open the Query engines pillar in depth order → · Depth tier for ch5 (Performant)
- Start here OrientationDurable
Push vs pull query engines.
Vectorized execution models for security analytics. Push engines win on throughput; pull engines win on adaptive replanning. What the difference means for your workload mix.
Read →
- DepthDurable
DuckDB for analyst-driven threat hunting.
Single-node SQL on parquet that an analyst can run from a laptop. Where DuckDB fits as the analyst-overlay tier, what it doesn't replace, and how the Lambda-native pattern works for ad-hoc hunts.
Read →
- DepthEvidence-pinned
dbt for security data.
Transformation patterns for detection engineers. The dbt-Sigma-OCSF model layer, where dbt is a clean fit, and where it competes with your pipeline tooling.
Read →
- EvidenceEvidence-pinned
One engine in front: StarRocks over shared Iceberg.
The strongest one-engine answer to the question a SOC actually asks: who is going to run four engines. StarRocks alone over the same Iceberg tables covers three of the four analyst-facing workloads from one MySQL-wire endpoint, and the scored join bench (Tier B, single host) prices the giveaways: no SOC-shaped join exceeded 1.5 s on any engine, so the spread that justified four endpoints is single-digit. The Splunk DB Connect leg, Tier D at first publication, has since been run: the stock MySQL driver connects and returns correct rows, a connectivity check rather than a migration-scale measurement.
Read →
- EvidenceEvidence-pinned
ClickHouse at petabyte scale.
Netflix's 5 PB/day ClickHouse optimization journey, from fingerprinting (216 µs → 23 µs) through native protocol serialization to tag sharding (3 s → 700 ms), plus Huntress's 90%+ cost reduction migrating from Elastic.
Read →
- EvidenceEvidence-pinned
Materialized views for security data.
What actually works at petabyte scale. Where MVs are the right answer (recurring dashboards, Sigma correlations), where they fall apart (high-cardinality, schema churn), and which engines deliver the lifecycle math.
Read →
- EvidenceEvidence-pinned
Isolation-first security data platforms.
Co-tenant scheduled detection load, rather than raw data volume, is what breaks the interactive query experience. A first-party single-host bench finds the knee at 64x the base scheduled rate and shows why a dedicated, isolated deployment is the failure you are paying to avoid.
Read →
12 essays
Pipelines & streaming.
Cribl, Tenzir, Vector. Kafka, NATS, streaming-database decisions. Where pipeline lock-in moved after the SIEM lock-in eased.
Open the Pipelines & streaming pillar in depth order → · Depth tier for ch4 / ch5 / ch7
- Start here OrientationEvidence-pinned
Cribl vs Tenzir vs alternatives.
Choosing your security data pipeline. The procurement-evidence-vs-OCSF-fidelity split, where Vector fits as the open-source third option, and what the v1 Capability Matrix puts at #1 across three archetypes.
Read →
- DepthEvidence-pinned
The pipe layer: what's missing from your AI security platform.
DSPM, DLP, and AI security platforms promise sub-second enforcement on the assumption that the data underneath is already normalized, low-latency, and lineage-aware. The pipe layer is the gap that decides whether they deliver, with Tenzir as the worked example of an OCSF-native pipe.
Read →
- DepthEvidence-pinned
Vector: the data router Datadog open-sourced.
Vector at the Archetype-C #1 spot for cost-and-lock-in-led shops. What VRL fluency costs, where Datadog stewardship raises trust questions, and the OCSF gap.
Read →
- DepthDurable
Pipeline lock-in.
Where switching costs moved next. The SIEM lock-in eased; the pipeline-tooling lock-in took its place. Which patterns reduce it; which vendors aggravate it.
Read →
- DepthDurable
The parsing layer nobody owns.
Security data quality breaks at the boundary between vendors, time most of all, and no one in the chain is paid to fix it. A first-hand case from a Palo Alto Splunk-app pull request, Zeek timestamps, and Tenable's nested data, and the argument for a fair broker.
Read →
- DepthEvidence-pinned
Observability pipelines and the security overlap.
Datadog OP, Edge Delta, and the boundary question. Where observability pipeline tooling does the security job credibly, and where the security workload still needs purpose-built tools.
Read →
- DepthDurable
ETL vs ELT for security data.
Who owns the schema, and when. The shift from upstream-normalized ETL to downstream-normalized ELT, what it costs in storage, and what it earns in flexibility.
Read →
- DepthDurable
Kafka architecture deep-dive.
Kafka as the security-data stream bus. Partition design, retention, the broker-replication math, and where it leaks operational complexity at scale.
Read →
- DepthEvidence-pinned
Kafka to Iceberg: the integration hidden costs.
Streaming Kafka into Iceberg looks straightforward and isn't. Connector quality, schema evolution under load, exactly-once semantics, and the small-files problem.
Read →
- DepthEvidence-pinned
The streaming database decision.
RisingWave, Apache Fluss, Kafka plus Iceberg, and Spark Real-Time Mode are four different bets on stateful streaming. Where each earns its keep for security workloads and where the batch lakehouse is still the right call.
Read →
- DepthEvidence-pinned
NATS JetStream: lightweight Kafka alternative, disqualified.
NATS JetStream looks like a Kafka simplification for security workloads, but the durability and retention story disqualifies it for that job. What it's actually good for instead.
Read →
- EvidenceDurable
Pipeline-based detection in stream processing.
Detection at the pipeline tier, before the lake. When it's appropriate, what it costs in operational complexity, and the trade-off against retroactive lake-side detection.
Read →
17 essays
Detection & hunting.
Detection-engineering maturity ladder, MLOps for hunters, latency tiers, feature stores, PEAK methodology on a modern data stack.
Open the Detection & hunting pillar in depth order → · Depth tier for ch3 (Trustworthy) and ch6 (what good looks like)
- Start here OrientationDurable
The ground you're already standing on.
Detection engineers are doing data modeling every day without naming it, and a rule can compile clean and quietly never fire because one of those mappings was wrong with nothing set to catch it. The vocabulary, the silent-failure mode, and a deductive check that caught 100% of injected type-crossings on a 925-row corpus.
Read →
- DepthDurable
Detecting the OT you can't parse.
Most industrial protocols will never get a deep parser, so the question that decides whether you can monitor an OT network is how much you can detect from behavior (timing, flow, who talks to whom) before a parser ever runs. Why behavioral-first gives immediate east-west coverage across the long tail, why it fits the NERC CIP-015-1 internal-monitoring mandate, where it stops, and the falsifier that would change my mind. It's a Tier B practitioner argument, without a benchmark behind it.
Read →
- DepthDurable
What your data means vs what shape it is.
Schema conformance checks the shape of a field and says nothing about its meaning, so a mapping can fit OCSF exactly and still mean the wrong thing. Eight flagged meaning-crossings from a six-schema crosswalk corpus (plausible coarse mappings worth a human review, not confirmed-wrong rows), made visible by a deductive check the syntax pass waves through.
Read →
- DepthDurable
Catching the mistake that kills a detection.
The wrong field mapping that silently kills a detection is catchable with a reasoner and a few disjointness assertions D3FEND never shipped. How to run that check yourself against your own mappings, and the limits of what it catches.
Read →
- DepthDurable
Who actually does the hunting.
The analyst reverse-engineering a broken field mapping at 2am is doing applied ontology, and naming the work hands them open tooling and a community that's been building the same map from the other side.
Read →
- DepthEvidence-pinned
The open detection-grounding stack, piece by piece.
A practitioner's map of the open detection-grounding stack: OCSF, D3FEND, Sigma, the ROBOT/ELK reasoner toolchain, and the six-schema crosswalks, with what each piece is for and exactly where it's thin.
Read →
- DepthDurable
The hunting maturity ladder: why HMM2 to HMM3 is the hard step.
David Bianco's Hunting Maturity Model read as an analyst-trust force multiplier: why most hunting programs sit at HMM2, why the step to HMM3 is the hardest and can't be bought, and why detection content authored at HMM3 survives a platform migration.
Read →
- DepthEvidence-pinned
PEAK and the lakehouse.
How modern data stacks enable threat hunting. The Splunk PEAK framework ported to lakehouse infrastructure, what works, and what doesn't.
Read →
- DepthEvidence-pinned
Three latency tiers: detection, hunting, analysis.
Detection in seconds, hunting in minutes, analysis in hours, separated by who is waiting on the result. The three-tier frame behind DetectFlow, why the 2026 breakout-time, time-to-exploit and Mythos data made detection latency a hard constraint, and the architecture that serves all three.
Read →
- DepthEvidence-pinned
MLOps tools for threat hunters.
Reproducibility for threat-hunt notebooks. What MLOps tooling brings to hunters that they don't already have, and where the friction lives.
Read →
- DepthEvidence-pinned
Where detection-as-code notebooks should live.
The notebook is how security analytics gets shared, and the live decision is the proprietary-but-production Databricks notebook versus the open, reproducible marimo .py. Why the lock-in moved up to the authoring layer, what marimo provably fixes, and why the detection you ship is text either way.
Read →
- DepthDurable
Every security tool ships with a silent configuration gap.
Manageability as a force multiplier: the distance between how a control is configured today and how it should be run, made visible and rankable across the whole stack. Threat-informed reallocation and CIS-benchmark conformance scanning optimize different denominators, so the ranking weight I'd use is threat exposure against an independent baseline rather than raw configuration distance.
Read →
- EvidenceEvidence-pinned
The assurance gap no single tool closes.
No single security tool knows your environment: on a planted 140,000-cell benchmark the best single tool recovers 47.7% of the truth and a freshness-scored cross-tool merge recovers 75.6%, while a 24.4% residual stays dark no matter how many tools you own. The transferable claim is the ordering, which holds across the whole parameter sweep, plus the entity-resolution tax a contested join key adds.
Read →
- EvidenceEvidence-pinned
The query engine returned the wrong answer and didn't tell you.
Over an open format the engine is supposed to be interchangeable. In my lab one returned a filtered count tens of rows short of the others on the identical Parquet, no error raised, and a timing-only benchmark would have published it as a win. Why a cross-engine answer-equality gate is the only thing that caught it.
Read →
- EvidenceEvidence-pinned
The better the model, the quieter the wrong answer.
I answered one OCSF question battery six ways across three model tiers. On compute-over-population questions no LLM-authored arm was reliably correct, and the more capable model made the wrong answer quieter rather than rarer; an execution loop and self-consistency both failed the same way. Safety lived in the layer that refuses a query it can't answer.
Read →
- EvidenceEvidence-pinned
Parquet doesn't hash the way your chain-of-custody thinks it does.
Chain-of-custody, WORM, and dedup workflows key on a file hash, but Parquet isn't byte-reproducible by default: parallel row-group order makes the same logical data produce a different SHA-256 every write. A faithful re-export breaks the hash and looks like tampering. Force determinism (threads=1 or ORDER BY, ~20% smaller too), and hash the logical content so a faithful re-export still matches.
Read →
- EvidenceEvidence-pinned
Measuring the detection coverage you actually have.
A coverage read from what actually fired rather than a vendor heatmap: eight ATT&CK techniques run through Sigma over an OCSF store on one APT29-style chain, scored against planted ground truth at 3 detected, 1 noisy, 4 missed, with the D3FEND join read narrowly, since countering a technique isn't detecting it.
Read →
4 essays
Migration & federation.
Migrating 800 detection rules across seven parallel ingest buses. Hidden costs. The federated rollout playbook. Splunk Federated Search as bridge or lock-in extension.
Open the Migration & federation pillar in depth order → · Depth tier for ch7 (Modularity and migration)
- Start here OrientationDurable
The hidden cost of SIEM migration.
The $300K project that became $1.2M. On the migrations I've scoped, the labor to move detection content, run two platforms in parallel, and retrain analysts ran 40–100% above the technology-only estimate. With a phased funding playbook your CFO will approve.
Read →
- DepthDurable
The federated rollout playbook.
Federated detection at scale across business units and clouds. The ordering that works, the common mistakes that compound, and the governance discipline that keeps it coherent.
Read →
- DepthEvidence-pinned
Splunk Federated Search: bridge or lock-in extension?
Splunk's federated-search story positioned as the bridge to a lakehouse. What it actually does, where it's a credible bridge, and where it's a lock-in extension.
Read →
- EvidenceEvidence-pinned
Migrating 800 detection rules across seven parallel ingest buses.
The conversion playbook from one detection rule corpus to another while seven ingest pipelines run in parallel. What automation closes, what it doesn't, and how to phase the cutover.
Read →
10 essays
Economics & measurement.
The cost optimization paradox in security data, the storage-media economics under the bill, and the cloud-versus-on-prem case for security telemetry. How to run a benchmark that doesn't lie, including the license clauses that leave vendor benchmarks as the only ones published.
Open the Economics & measurement pillar in depth order → · Depth tier for ch1 (Manageability) and ch3 (benchmarking honesty)
- Start here OrientationDurable
The two cultures: evidence and marketing.
Data engineering built a culture of evidence (public benchmarks, open architectures, documented failures, quantified trade-offs) while security decides on vendor marketing, analyst reports, anecdote, and compliance checklists. The five principles of modern data architecture framed for a security reader, and what they become when you build them for security.
Read →
- DepthDurable
We are our own worst enemy.
Reed's Law says a set of N members can form 2^N − N − 1 subgroups, and read backwards over a security stack that is the bill nobody budgets, because what you end up paying for is the combinations between the tools. Walks the lineage from Schneier and Geer through Perrow and Conway, takes Venables's objection that bad design is the real enemy at full strength, and answers it, since the design failure is collective and combinatorial rather than lazy. Ends on the two fixes: an open narrow waist in the stack, and what the same logic asks of a new domain owner.
Read →
- DepthEvidence-pinned
Why modern data stacks haven't replaced SIEMs.
The technical case for the modern data stack is settled and the SIEM persists anyway, because the barrier was never technical. The four real barriers to leaving (risk aversion and 2am blame, a skills-timeline mismatch, the migration rebuild itself, status-quo bias), why the team defending the SIEM is a rational actor managing operational risk, and what actually moves it.
Read →
- DepthEvidence-pinned
The write endurance security data never spends.
Drive media is the majority of a security data platform's bill, which makes the NVMe endurance tier a first-order cost decision. The contrarian read: mixed-use and write-intensive drives are sold at a premium a write-once-read-rarely security lake almost never consumes, and the industry stopped publishing the data that would let you check.
Read →
- DepthEvidence-pinned
The repatriation case for security data.
Cloud repatriation is well-covered macro ground. The narrower, stronger claim: security telemetry is the textbook workload to bring home, because it's steady-state, write-heavy, retained for years and increasingly bound by sovereignty rules, which is exactly the profile cloud's variable pricing overcharges. The cheap, correctly-specced on-prem media is what closes the math.
Read →
- DepthEvidence-pinned
The index in your bucket.
A tip said a category called 'scanners' was having a moment. The category name turned out not to exist, the architecture is real and older than its supposed leader, and between index-as-store and open-table sits a third build nobody has fully priced: raw logs in your bucket at S3 economics, searchability in a proprietary sidecar only the vendor's engine can read. Where the lock-in actually goes at exit, and the correctness question a closed index makes unanswerable.
Read →
- DepthDurable
How to run a benchmark that doesn't lie.
Rule zero, learned the hard way: verify the answer before you trust the clock, because one engine returned a filtered count tens of rows short over byte-identical Parquet and a timing-only run would have published it as a win. The rest of the method follows from that: report the CV, scale until the signal clears the noise, register identical bytes, isolate the run, control the power plan, hash logical rows not file bytes.
Read →
- DepthDurable
The storage is moving, the engine isn't (yet).
The loud off-Splunk story says the open lakehouse is replacing the SIEM analytics engine at scale, now. The quieter, better-sourced read: storage and pipeline are moving fast and measurably while the engine stays put, and the gap between the two is where a buyer gets oversold.
Read →
- DepthEvidence-pinned
The cost optimization paradox in security data.
Practitioner surveys rank cost optimization near the bottom of the priority list while boards and markets force it first. The collision between those two timescales, and why it keeps producing the same SIEM cost conversation even as the tooling around it improves.
Read →
- EvidenceEvidence-pinned
The index pays twice.
A hot search index stores 4.2× the bytes at 3.5× the price per byte, so it costs about 14.8× a warm Iceberg-on-S3 lakehouse for the same events. At thirty days the gap is a rounding error; at the seven-year retention horizon a regulated firm has to plan for, it is the difference between a line item and a project. A measured storage floor, Tier B and first-party single-host, explicitly not a TCO model.
Read →
7 essays
AI, automation & vendor watch.
Two kinds of writing share this pillar. The first is securing the AI you deploy, worked from the standards and the lab rather than from vendor framing: the lethal trifecta now that ATT&CK v20 gives two of its legs their own technique IDs, who keeps custody of the forensic data when a hosted model refuses the incident work, and the case for scoring security tools on whether you can run, inspect and own them. The second is direction-tracking, where the agentic-SOC, semantic-layer and AI-maturity claims get read against an evidence tier to map where the field is heading, without being claimed as core thesis. The security-data measurements stay the anchor for both.
Open the AI, automation & vendor watch pillar in depth order → · Vendor-watch layer, tracked to read direction; no book chapter by design
- Start here OrientationDurable
The lethal trifecta, explained three ways.
One AI-agent risk told three ways. Simon Willison's lethal trifecta (private data, untrusted content, a way out) is now two named ATT&CK v20 techniques, T1698 and T1700. Users get habits, builders get a remove-one-leg design rule walked through two ATLAS case studies, and executives get a generic usage-metrics template.
Read →
- DepthDurable
When the frontier model refuses the forensics
In July 2026 the hosted frontier models Hugging Face reached for first refused much of its incident forensics, so the team rerouted the pipeline through an open-weight GLM-5.2 build on its own hardware. Why a well-tuned usage policy still fails the defender, why custody should come first, what self-hosting costs and where the local path trails the frontier, and the platform requirement that follows: the model is a component you can swap under load.
Read →
- DepthDurable
Security has an ontology. It doesn't have a semantic layer.
D3FEND settled security's knowledge layer; nothing binds that knowledge to tables and metrics. The schemas carry no metrics, the metric catalogs aren't machine-readable, the coverage formats exchange scores with no defined derivation, and the commercial products that do compute metrics publish no standard. Apache Ossie (incubating) is the first credible candidate to close the gap, verticals are organizing inside it now, and security isn't in the room.
Read →
- DepthDurable
The security market has no one defining what you can own.
Vendors could ship security tools you can run, inspect, and air-gap, but nothing makes them, and no independent force scores whether a given tool lets you own it. The agentic rush is widening the gap. Sigma is the proof the open pattern wins when it has a champion; vendor MCP servers are the tell; the missing piece is a referee with a practitioner-ownability axis.
Read →
- DepthDurable
The Gatsby Summer of AI.
AI maturity read through early-automotive history: past the horseless-carriage stage, building AI-native, but in a chaotic pre-seatbelt era where capability outran the safety infrastructure. The bill the glamour hides shows up when you measure it: NL2KQL runs clean 97–99% of the time and returns the correct result set only about 58%.
Read →
- DepthDurable
Agentic analysis reinvented Kimball. It skipped the measurement.
The 2026 consensus that AI agents need dimensional modeling, semantic layers, and governed ontologies is mostly right, though it's quiet about the one step that matters when the data is evidence. Autonomously-built mappings fail sound-but-wrong and silently; the answer is a deductive check that fails on a mapping that looks correct, model-independent, where a human review pass structurally can't. The AI-generated-OCSF-parser claim is the cleanest worked example: transformative only at production-grade accuracy.
Read →
- DepthEvidence-pinned
What's real vs marketed in the agentic SOC.
Claims about agentic security-data deserve a definition and an evidence tier before anyone repeats the headline. The practitioner ceiling sits at 30-40% end-to-end against the 90%+ the slides claim, and the binding constraint is cross-vendor identity and trust (MIT's NANDA), which matters more than the automation percentage, while the pipeline layer is where the migration risk consolidates. What ships, what doesn't, and what to plan for.
Read →
Notifications
Get a note when a new essay or benchmark publishes.
It's low-volume: essays and benchmark results as they're ready, with nothing else and no drip campaigns.
The hypothesis-grounded work (twelve anchor hypotheses with evidence tiers, twenty-nine contradictions tracked over time, and the method-in-practice essay) lives on the research page. The program POV that connects them is on thesis.