Writing · Pillar
Pipelines & streaming.
Cribl, Tenzir, Vector. Kafka, NATS, streaming-database decisions. Where pipeline lock-in moved after the SIEM lock-in eased.
Depth tier for ch4 / ch5 / ch7 · 12 essays
Listed from the front door down to the measured evidence. Start here marks the entry essay. The reading-level tag says whether you are about to read the frame (Orientation), the argument (Depth), or the narrowest first-party finding (Evidence). The durability tag flags whether a number is durable architecture or an Evidence-pinned result tied to a version or price and meant to be re-run. The last tag is the provenance state, meaning who held the pen and who checked the work, which is explained in full on how these are made.
- Start here OrientationEvidence-pinnedAI-drafted, observed
Cribl vs Tenzir vs alternatives.
Choosing your security data pipeline. The procurement-evidence-vs-OCSF-fidelity split, where Vector fits as the open-source third option, and what the v1 Capability Matrix puts at #1 across three archetypes.
Read →
- DepthEvidence-pinnedAI-drafted, observed
The pipe layer: what's missing from your AI security platform.
DSPM, DLP, and AI security platforms promise sub-second enforcement on the assumption that the data underneath is already normalized, low-latency, and lineage-aware. The pipe layer is the gap that decides whether they deliver, with Tenzir as the worked example of an OCSF-native pipe.
Read →
- DepthEvidence-pinnedAI-drafted, observed
Vector: the data router Datadog open-sourced.
Vector at the Archetype-C #1 spot for cost-and-lock-in-led shops. What VRL fluency costs, where Datadog stewardship raises trust questions, and the OCSF gap.
Read →
- DepthDurableAI-drafted, observed
Pipeline lock-in.
Where switching costs moved next. The SIEM lock-in eased; the pipeline-tooling lock-in took its place. Which patterns reduce it; which vendors aggravate it.
Read →
- DepthDurableAI-drafted, observed
The parsing layer nobody owns.
Security data quality breaks at the boundary between vendors, time most of all, and no one in the chain is paid to fix it. A first-hand case from a Palo Alto Splunk-app pull request, Zeek timestamps, and Tenable's nested data, and the argument for a fair broker.
Read →
- DepthEvidence-pinnedAI-drafted, observed
Observability pipelines and the security overlap.
Datadog OP, Edge Delta, and the boundary question. Where observability pipeline tooling does the security job credibly, and where the security workload still needs purpose-built tools.
Read →
- DepthDurableAI-drafted, observed
ETL vs ELT for security data.
Who owns the schema, and when. The shift from upstream-normalized ETL to downstream-normalized ELT, what it costs in storage, and what it earns in flexibility.
Read →
- DepthDurableAI-drafted, observed
Kafka architecture deep-dive.
Kafka as the security-data stream bus. Partition design, retention, the broker-replication math, and where it leaks operational complexity at scale.
Read →
- DepthEvidence-pinnedAI-drafted, observed
Kafka to Iceberg: the integration hidden costs.
Streaming Kafka into Iceberg looks straightforward and isn't. Connector quality, schema evolution under load, exactly-once semantics, and the small-files problem.
Read →
- DepthEvidence-pinnedAI-drafted, observed
The streaming database decision.
RisingWave, Apache Fluss, Kafka plus Iceberg, and Spark Real-Time Mode are four different bets on stateful streaming. Where each earns its keep for security workloads and where the batch lakehouse is still the right call.
Read →
- DepthEvidence-pinnedAI-drafted, observed
NATS JetStream: lightweight Kafka alternative, disqualified.
NATS JetStream looks like a Kafka simplification for security workloads, but the durability and retention story disqualifies it for that job. What it's actually good for instead.
Read →
- EvidenceDurableAI-drafted, observed
Pipeline-based detection in stream processing.
Detection at the pipeline tier, before the lake. When it's appropriate, what it costs in operational complexity, and the trade-off against retroactive lake-side detection.
Read →
