Architecture deep-dive
The pipe layer: what's missing from your AI security platform.
Every DSPM, DLP, and AI security platform I evaluate makes the same architectural assumption: that the data underneath is already normalized, low-latency, and lineage-aware by the time the platform sees it. In most organizations I've worked with, that assumption is false. The pipe layer (the part of the stack responsible for getting telemetry into a queryable shape in real time) is the gap that determines whether the platform above can deliver what it promises.
Reading time: about 18 minutes. Evidence tier: B overall (vendor documentation, CEO framing, production deployment claims), with several Tier B+ third-party validations and one Tier D speculative direction. Performance numbers from vendors are flagged where I cite them, and I have not independently benchmarked 100+ Gbps streaming claims against security-shaped data.
The framing
Policy layer versus pipe layer.
Matthias Vallentin, Tenzir's CEO, has a one-line version of this argument that's worth quoting because it captures the asymmetry I see across the AI security market: "Everyone's building the policy layer. Nobody's talking about the pipes." That's vendor framing and I treat it as Tier B evidence, but the shape of the observation matches what I find in customer architectures.
The policy layer is the part of the stack security teams have been buying aggressively for the last three years, and it covers a lot: DSPM (data security posture management) platforms like Cyera and Securiti, DLP (data loss prevention) platforms like Netskope, AI security posture management offerings from Palo Alto and others, and SIEM detection rule packs. All of them are doing legitimate work, whether that's classification, policy enforcement, alerting, or posture tracking, but none of that work can happen until the underlying telemetry is in a shape the policy engine can consume.
The pipe layer is the part of the stack that gets the telemetry into that shape, which means streaming ingestion at the throughput your environment produces, schema normalization at ingest rather than as a batch post-process, intelligent routing so the expensive SIEM only gets the data that justifies the cost, and lineage tracking so compliance auditors and ML model decisions can both be defended later. The pipe layer is unglamorous, but it's the foundation that everything above it depends on.
My argument here isn't that Tenzir is the right pipe layer, but rather that any AI security platform without a real pipe layer underneath is making claims the infrastructure can't honor. Tenzir is the named example I'll work with because the company is most explicit about positioning at this layer, though the analysis applies just as well to Cribl, Vector, DataBahn, and whatever else surfaces over the next two years.
Vendor claims vs reality
Three platforms, three mismatches.
The clearest way to see the pipe-layer gap is to take the policy-layer marketing claims at face value and ask what infrastructure each one implicitly requires, so here are three examples from the current AI security market.
Netskope: sub-100ms DSPM latency
Netskope markets real-time DSPM discovery with sub-100ms latency for data classification and policy decisions, and to deliver on that number end-to-end, the platform needs normalized telemetry (OCSF or equivalent) arriving in a streaming pipeline with sub-second latency from the source. If the data is older than that, then the "sub-100ms" measurement is really from pipe output to policy decision, while the journey from raw telemetry to enforcement remains unmeasured.
In most organizations I've audited, the upstream pipeline is batch ETL: Logstash to Kafka to Spark with hourly or daily aggregation, then a normalization step, then a load into the DSPM platform. The sub-100ms claim is technically accurate at the platform boundary, but it becomes irrelevant for any real-time enforcement story when the data arriving at that boundary is already several hours old.
Cyera: real-time AI Guardian for DLP
Cyera positions an AI-powered real-time data loss prevention layer with classification and enforcement in the loop. To deliver real-time enforcement, the platform needs streaming OCSF normalization at ingest (not as a downstream batch step), real-time enrichment so the model knows which data is sensitive before the policy decision, and immediate policy hooks rather than post-processing. The common failure mode is the second of these, because most organizations normalize in batch pipelines, so "real-time AI Guardian" becomes, in practice, an after-the-fact reviewer that may still be useful for investigations even though it operates as review rather than enforcement.
Palo Alto: unified AISPM
Palo Alto's AI security posture management (AISPM) markets a unified view of AI security across the organization, covering cloud workloads, on-prem applications, endpoints, model inference paths. To deliver a unified posture, the platform needs columnar data pipelines with consistent lineage across sources, a metadata layer that survives schema migrations, and multi-source data fusion that doesn't break when one upstream source changes its event shape. Schema drift across fragmented pipelines is the most common reason "unified" posture tracking degrades over time, and because that drift originates below the policy layer, it's a gap no policy-layer platform can close on its own.
What a pipe layer does
Four responsibilities, not negotiable.
When I sketch what a real pipe layer needs to do for an AI security platform to deliver on its marketed capabilities, the list is short and stubbornly specific, and these are the four responsibilities that, in my experience, separate a pipe layer from a glorified log forwarder.
1. Streaming ingestion at environment-relevant throughput
The pipe layer needs multiple protocols (syslog, Kafka, S3 events, HTTP APIs, agent-generated structured logs) converging into a single streaming runtime, and it needs zero-copy transformations using Apache Arrow buffers so the serialization tax I describe in the Arrow and ADBC piece doesn't re-appear here. It also needs to be streaming-first rather than micro-batch, because micro-batch is where the latency story silently breaks for AI security workloads.
The throughput number matters, but only in context. A 5,000-person enterprise running comprehensive telemetry produces 1–10 TB/day, which is well under 1 Gbps sustained. A 100,000-person enterprise running comprehensive logging may peak at 50–100 Gbps during incidents. Tenzir markets 100+ Gbps ingestion, which is Tier B evidence (vendor claim, no independently audited benchmark I've found), and I'd treat it as a directional capacity claim rather than a number a procurement decision should rest on. So the right question for an architect isn't whether the pipe can do 100 Gbps, but whether the pipe handles your P99 burst without dropping events. My own ingest bench puts a measured floor under that question rather than a marketed ceiling: on a single host (sdw-lab, Tier B) Tenzir sustained about 89.6k events/sec, in the same range as rsyslog at 93.7k and well ahead of Vector at 26.1k, but the result that actually matters for a procurement decision is the reliability one — the Tenzir 6.0.0 static build segfaulted on roughly one run in six at the 1M-event mark, a cliff you find by measuring sustained behavior, not by reading a peak-throughput number off a datasheet.
2. Real-time schema normalization at ingest
Here the pipe layer needs native OCSF (Open Cybersecurity Schema Framework) mapping with automatic
field detection, streaming normalization rather than post-processing, and schema-on-read flexibility
for the long tail of vendor formats that haven't yet been mapped. The reason this is non-negotiable is
that every downstream
policy layer assumes a single, stable schema, and every fragmented pipeline that normalizes per
source produces schema drift that eventually breaks the assumption. Centralizing normalization at
the pipe layer is the only way I've seen this stay tractable over a multi-year deployment, with the
normalization contract held fixed and the router behind it left replaceable (the ./moar swap-router check in docker/, with the OCSF mapping check in gate/).
I ran that check on the MOAR reference stack on 2026-06-07 (single host, ./moar swap-router):
the same raw Okta event routed to OCSF Authentication through Vector, Tenzir, and Fluent Bit produced the
identical OCSF Authentication contract across all three routers, down to class_uid 3002, the
activity_id of 1 or 2, and the same user and src_ip fields. That's the
point of centralizing the contract at the pipe layer rather than per router: the normalization output is
the fixed point an architect can design against, while the router that produces it stays swappable, so a
later migration from one engine to another doesn't ripple a new schema into every downstream policy
platform. I want to be precise about what this is, because it's a contract-equality check on a single host
(a portability proof that three routers agree on the OCSF output), not a throughput measurement, and the
Tenzir 100+ Gbps capacity number stays a vendor-reported claim I haven't audited.
3. Intelligent routing
Here high-signal events go to the SIEM where the licensing cost is justified, bulk telemetry goes to the lake where it's cheap to retain, and threat intelligence enrichment happens once, at the routing layer, rather than independently in three downstream platforms. Cribl is the most production-validated example of this responsibility, because the company built its business on "route, reshape, reduce" and the third-party validation (RiverSafe's ~40% reduction in SIEM ingest is one example I've seen cited) is real. Tenzir markets the same routing capability with a stronger emphasis on Arrow-native transformation and TQL (Tenzir Query Language) as the routing-rule surface.
4. Lineage tracking
Here the pipe layer needs metadata that survives schema migrations, along with the ability to point at a specific detection decision and trace it back to the raw telemetry, the enrichment steps, the normalization version, and the routing rules in effect at the time. This is the responsibility that has the least vendor marketing attention and, in my view, the most compliance-driven future demand, because ML-driven detection without lineage isn't auditable, and an unauditable AI security platform is one regulator audit away from becoming a procurement crisis.
The anti-pattern
Batch ETL pretending to be a pipe layer.
The architecture I see most often, and the one I'd argue creates the largest gap between AI security platform marketing and operational reality, looks roughly like this: Zeek and Suricata feed Logstash, which writes to Kafka, which is consumed by a Spark batch job that aggregates and normalizes on a 15-minute or hourly cadence, which writes Parquet to S3, which is then registered into the DSPM or DLP platform. Every box in that diagram is doing legitimate work, so the problem isn't any single box but the aggregate latency they add up to.
Four specific failure modes show up in this pattern. The DSPM platform is making decisions on data that's hours old, which means real-time enforcement is structurally impossible regardless of what the platform's internal latency benchmarks say. There's no single normalization point, so schema drift across the Logstash configs, the Spark job, and the Parquet writer creates inconsistencies that the downstream consumer has to handle case-by-case. There's no lineage metadata, so when an ML model in the policy layer flags a decision, no one can trace back to the raw events and the enrichment snapshot that produced it. And the Spark batch cost is non-trivial, since you're paying for batch compute to do normalization work that should happen once, in-stream, at the pipe layer.
The streaming alternative replaces the middle of that diagram with a single streaming fabric: Zeek and Suricata feed the pipe layer directly, which performs OCSF normalization in-stream, applies enrichment, routes high-value events to the SIEM and bulk telemetry to Iceberg, and emits lineage metadata alongside the data. The downstream DSPM, DLP, or detection platform sees data that's seconds old rather than hours old, with consistent schema and traceable lineage.
I want to be specific about what this is, because it isn't a claim that streaming is always better than batch. Batch processing is correct for many security workloads, especially long-horizon analytics and compliance reporting. The narrower claim I'm making is that the specific responsibilities that make AI security platforms work are streaming responsibilities, so trying to discharge them in batch ETL is the most common reason the platforms underdeliver.
The vendor landscape
Who is building at the pipe layer.
There are four vendors I'd put on a serious evaluation list today, each occupying a different point in the design space, and I'm naming them because vague references to "the pipe layer market" hide the actual trade-offs.
Tenzir
Tenzir is the vendor that markets most explicitly at the architectural layer I describe in this piece. The product centers on streaming ingestion with native OCSF support, Apache Arrow as the internal data shape, and TQL as the transformation and routing language. Differentiation against Cribl, in Tenzir's own framing, is the columnar-native internal pipeline and the schema-first approach to normalization. Evidence tier is B: Vallentin's CEO framing plus production deployment claims, no independently audited benchmark I've seen against security-shaped telemetry.
What I'd flag for an architect evaluating Tenzir today: the architectural claim is strong, the production reference list is smaller than Cribl's, and the 100+ Gbps capacity claim should be validated against your actual workload before a procurement decision rests on it. The Cribl vs Tenzir piece works through the head-to-head comparison in more detail.
Cribl
Cribl is the production-validated pipe layer for the cost-optimization use case. The "route, reshape, reduce" pattern reduces SIEM ingest cost by routing low-value telemetry to a cheaper store before it hits Splunk. Cribl Search adds a federated query capability across the routed data. Evidence tier is A on cost reduction (third-party validation, including RiverSafe's ~40% SIEM-ingest reduction reference, plus a long list of customer references). What Cribl doesn't aggressively market (and where I'd push on the architecture) is the schema-normalization and lineage-tracking responsibilities, because routing is the strongest piece while the rest is improving without being the headline.
Vector (Datadog)
Vector is the open-source pipe-layer tool that came out of Timber.io and is now stewarded by Datadog. It's high-throughput, written in Rust, and widely deployed in observability stacks. The honest framing for security architects: Vector handles the routing and transformation responsibilities well, doesn't have first-class OCSF support out of the box, and gets you off the Logstash treadmill if you're stuck there. It's the right answer when the problem you're solving is "I need a faster, more reliable pipe than Logstash," though it's a poorer fit when the problem is "I need a security-specific schema and lineage layer." The Datadog acquisition (2021) hasn't materially changed Vector's posture as an open-source project, though long-term governance is worth watching.
DataBahn
DataBahn is the newer entrant marketing AI-native security data fabric: automated parsing, schema drift detection, OCSF normalization, and pre-SIEM routing. The pitch is closest to Tenzir's in shape, with more emphasis on "AI agents handle parsing changes" as the differentiator. Evidence tier is C right now (vendor marketing, limited public production references in early 2026). I include DataBahn on the evaluation list because the category is moving quickly and the automated-parsing claim is the kind of capability that, if it holds up under real schema drift, materially changes the long-term cost of running a pipe layer. I would not bet an architecture on DataBahn today without proof-of-concept work against a representative slice of your telemetry.
Three use cases
Where a real pipe layer changes the architecture.
Agentless DSPM discovery across heterogeneous stores
The use case: run PII detection consistently across S3 buckets, Snowflake tables, and on-prem Postgres databases. Without a pipe layer, this becomes a separate scan per data store, each producing results in its own schema, batched on its own cadence, with no unified output for the DSPM platform to consume. With a pipe layer, the scans feed a single streaming fabric that normalizes results into OCSF, applies consistent classification logic, and emits lineage so an auditor can trace why a particular field was flagged as PII. The architectural difference is large, because the pipe layer is what turns three siloed reports into one unified posture view.
AI threat detection models with auditable decisions
The use case: deploy an ML model that flags anomalous user behavior, with the requirement that every flagged decision can be traced back to the inputs and enrichments that produced it. Without a pipe layer, the model trains on batch-enriched data that may not match the runtime enrichment, and the lineage required to defend a decision in a regulator audit doesn't exist. With a pipe layer that does real-time enrichment and lineage tracking, the model sees consistent inputs and every decision has a defensible trail. This is the use case where I'd argue lineage stops being a nice-to-have and starts being a compliance precondition, and where most current security data architectures fall short.
Multi-cloud security data unification
The use case: build a unified security data lake across AWS, Azure, and GCP. Without a pipe layer, each cloud has its own ingestion pipeline, its own schema mapping, and its own retention policy, and cross-cloud queries either don't work or require expensive ETL bridges. With a pipe layer that ingests from all three clouds into a single Arrow-native fabric and writes to a shared Iceberg lake, you get one schema, one retention policy, and cross-cloud queries that are no harder than single-cloud queries. This use case is where the pipeline lock-in piece does heavy lifting, since the pipe layer is also where multi-cloud independence is either preserved early or lost to per-cloud sprawl that's expensive to undo later.
The convergence
Why this matters more as DLP and DSPM merge.
Francis Odum's framing of the current AI security market is that AI security is becoming the control layer that merges DLP and DSPM into a single category. That's analyst commentary (Tier B), and I think the directional argument is correct: the boundary between "preventing data loss" and "managing data security posture" was always artificial, and AI-driven classification is the lever that collapses it.
The implication for the pipe layer is that the data layer underneath the merged control layer has to be unified too. You can't have one schema for DLP-classified events and a different schema for DSPM-classified events when the same policy engine is making decisions about both. The merged control layer requires a pipe layer that produces a single, consistent, lineage-aware view of all telemetry, and that requirement is what turns the pipe layer from a nice architectural cleanup into a structural precondition for the platform to work at all.
The honest counterargument is that the merger may not happen on the timeline analysts are projecting, or may happen with the merged category absorbing pipe-layer responsibilities into the policy platform itself, and both are possible. So the conservative architectural choice is to assume the pipe layer remains a distinct concern and to invest accordingly, because the cost of a clean pipe layer is much lower than the cost of retrofitting one underneath a policy platform that assumed it already existed.
Practical guidance
How to audit your pipe layer today.
There are four diagnostic questions that, in my experience, separate organizations whose AI security platforms will deliver from organizations whose platforms will produce expensive disappointment, and none of them require a new procurement, since they're audit questions you can run against your existing architecture this quarter.
- What's the end-to-end latency from telemetry generation to queryable state? Not the latency the policy platform reports, but the wall-clock time from a Zeek log line being written to that log line being available to a DSPM or detection query. If the answer is "we don't measure this," that's the first finding.
- Where does schema normalization happen? If it's distributed across Logstash configs, Spark jobs, and downstream connectors, you have schema drift waiting to surface. If it happens once, at a defined point in the pipeline, you have a real pipe layer at the normalization responsibility even if it's not branded as such.
- Can you trace a single detection decision back to its inputs? Pick a recent alert from your SIEM or DSPM platform. Can you produce the raw telemetry, the enrichments applied, the normalization version, and the routing path that produced it? If not, lineage is the gap, and it's the gap that becomes binding under regulatory scrutiny of ML-driven decisions.
- What happens at your P99 burst, not your average? Average throughput numbers are almost always tractable, but during incident-driven bursts the pipe layer either degrades gracefully or silently drops events, and the right time to find out which is during a planned test rather than during the incident that determines whether you have evidence later.
The order I'd run those questions in matters, because latency and schema normalization are tractable in existing architectures, so they're work but they don't require new vendors. Lineage and burst behavior are where vendor selection starts to matter, since retrofitting them onto a pipeline that wasn't designed for them is expensive.
Investment ordering
What to fix first.
If the audit surfaces gaps (which it will, in nearly every architecture I've evaluated), the investment order matters more than the choice of vendor. I'd prioritize as follows.
First, streaming ingestion with sub-second latency. This is the foundation everything else depends on. Without it, every downstream improvement is bounded by the worst-case latency of the slowest batch step. Vendor options range from Vector (open source, no security opinionation) to Cribl (production-validated, routing-first) to Tenzir (security-opinionated, Arrow-native), and you should pick based on what your team is going to operate rather than on whose marketing is most aligned with my framing.
Second, OCSF normalization at the ingest layer. Centralize the schema concern. Even if the policy layer above can handle multiple schemas, the operational cost of letting schema drift propagate compounds over time.
Third, lineage tracking. Build the metadata layer that lets every downstream decision be defended. This is the responsibility most likely to become a compliance requirement within two to three years, and the responsibility hardest to bolt on later. Iceberg V3's row-level lineage is one piece of the answer (the Iceberg vs Delta piece goes into detail), but it's not the whole answer at the pipe layer.
Fourth, intelligent routing. Cost optimization comes last not because it's unimportant (Cribl's customers will tell you it pays for the rest of the pipe layer), but because routing rules on top of an unreliable pipe just route unreliable data faster, so you want the foundation right before you optimize the economics.
Honest gaps
What I haven't validated.
Five things I'd flag for any architect using this analysis to make procurement decisions, because the gaps in my evidence matter as much as the framing.
- Vendor throughput claims are unaudited. Tenzir's 100+ Gbps number, Cribl's "10x queries at half the cost" line, DataBahn's automated-parsing claims: none of these have published methodologies I can audit. Treat them as directional capacity statements, not verified performance promises.
- I have not benchmarked pipe-layer products against OCSF-shaped data. The security-specific benchmark suite is on the lab roadmap and would be Tier B evidence at best when it lands. Until then, the analysis here is structural, not empirical.
- The DLP/DSPM convergence is analyst projection. Francis Odum's framing is plausible and matches the directional vendor moves I see, but the timeline could be wrong, or the merged category could absorb pipe-layer responsibilities rather than depend on them. Architectural hedging matters.
- Lineage as a compliance requirement is forward-looking. I'm projecting that regulators will eventually require auditable AI security decisions. That's a defensible projection based on the direction of EU AI Act enforcement and US sectoral regulation, but it's not yet a binding requirement in most jurisdictions.
- Multi-cloud pipe layers are still emerging. The vendors I name all support multi-cloud ingestion in principle, but the production references for cross-cloud security data lakes at scale are thin. This is a 2026 architectural argument, not a 2024 one.
None of these gaps undermine the structural argument, because the pipe layer is the bottleneck regardless of which vendor wins or which timeline holds. They do mean that the right way to use this analysis is as a framework for asking better questions during procurement, while the benchmarking against your own workload still has to happen separately.
Conclusion
Pipes before policy.
The AI security platform market is crowded at the policy layer. The pipe layer is where the infrastructure gap sits, and where most architectures I evaluate are betting their DSPM, DLP, and AISPM investments without naming the bet.
The argument I'd make to a security architect today is straightforward, because your policy platform's "sub-100ms latency" or "real-time AI Guardian" claims may be technically accurate at the platform boundary, but they can only be as real-time as the pipes that feed them. If the pipes are batch ETL with hourly aggregation, then the policy platform is doing post-facto review, which may still be valuable even though it isn't the enforcement the marketing implies.
The work, in order: audit your end-to-end latency, centralize schema normalization at ingest, add lineage tracking, and then optimize routing economics. Pick a vendor (Tenzir, Cribl, Vector, DataBahn, or whatever ships next) based on which set of responsibilities you weight most. The vendor choice matters less than the architectural decision to treat the pipe layer as a distinct concern that the rest of the stack depends on.
The policy layer is crowded while the pipe layer is the gap, so fixing the pipes is what lets the policy platforms deliver what they promise.