Writing · Pillar
Pipelines & streaming.
Cribl, Tenzir, Vector. Kafka, NATS, streaming-database decisions. Where pipeline lock-in moved after the SIEM lock-in eased.
Depth tier for ch4 / ch5 / ch7 · 12 essays
Listed from the front door down to the measured evidence. Start here marks the entry essay. The reading-level tag says whether you are about to read the frame (Orientation), the argument (Depth), or the narrowest first-party finding (Evidence). The durability tag flags whether a number is durable architecture or an Evidence-pinned result tied to a version or price and meant to be re-run. The last tag is the provenance state — who held the pen and who checked the work — explained in full on how these are made.
- Start here OrientationEvidence-pinnedAI-drafted, observed
Cribl vs Tenzir vs alternatives.
Choosing your security data pipeline. The procurement-evidence-vs-OCSF-fidelity split, where Vector fits as the open-source third option, and what the v1 Capability Matrix puts at #1 across three archetypes.
Read →
- DepthEvidence-pinnedAI-drafted, observed
The pipe layer: what's missing from your AI security platform.
Tenzir as the OCSF-native pipe layer. What it ships today, what the production evidence floor actually is, and where it's the upgrade path from Cribl.
Read →
- DepthEvidence-pinnedAI-drafted, observed
Vector: the data router Datadog open-sourced.
Vector at the Archetype-C #1 spot for cost-and-lock-in-led shops. What VRL fluency costs, where Datadog stewardship raises trust questions, and the OCSF gap.
Read →
- DepthDurableAI-drafted, observed
Pipeline lock-in.
Where switching costs moved next. The SIEM lock-in eased; the pipeline-tooling lock-in took its place. Which patterns reduce it; which vendors aggravate it.
Read →
- DepthDurableAI-drafted, observed
The parsing layer nobody owns.
Security data quality breaks at the boundary between vendors, time most of all, and no one in the chain is paid to fix it. A first-hand case from a Palo Alto Splunk-app pull request, Zeek timestamps, and Tenable's nested data, and the argument for a fair broker.
Read →
- DepthDurableAI-drafted, observed
Pipeline-based detection in stream processing.
Detection at the pipeline tier, before the lake. When it's appropriate, what it costs in operational complexity, and the trade-off against retroactive lake-side detection.
Read →
- DepthEvidence-pinnedAI-drafted, observed
Observability pipelines and the security overlap.
Datadog OP, Edge Delta, and the boundary question. Where observability pipeline tooling does the security job credibly, and where the security workload still needs purpose-built tools.
Read →
- DepthDurableAI-drafted, observed
ETL vs ELT for security data.
Who owns the schema, and when. The shift from upstream-normalized ETL to downstream-normalized ELT, what it costs in storage, and what it earns in flexibility.
Read →
- DepthDurableAI-drafted, observed
Kafka architecture deep-dive.
Kafka as the security-data stream bus. Partition design, retention, the broker-replication math, and where it leaks operational complexity at scale.
Read →
- DepthEvidence-pinnedAI-drafted, observed
Kafka to Iceberg: the integration hidden costs.
Streaming Kafka into Iceberg looks straightforward and isn't. Connector quality, schema evolution under load, exactly-once semantics, and the small-files problem.
Read →
- DepthEvidence-pinnedAI-drafted, observed
The streaming database decision.
Materialize, RisingWave, Flink SQL. Where streaming databases earn their keep for security workloads and where the batch lakehouse is still the right call.
Read →
- DepthEvidence-pinnedAI-drafted, observed
NATS JetStream: lightweight Kafka alternative, disqualified.
An honest disqualification. NATS JetStream looks like a Kafka simplification for security workloads; the durability and retention story disqualifies it. What it's actually good for instead.
Read →
