Lakehouse-native security data platform on the Databricks medallion architecture: security telemetry normalized to OCSF on the Silver layer, Unity Catalog as the governance plane, more than 21 billion security events queryable in under a minute. Atlassian built and operates it; this is SDW's reading of the public architecture (the Databricks customer story and the Data + AI Summit talk), not work SDW delivered.

21B+

Security events queryable in under a minute. Photon cut query times from 17 s to 5 s, retention extended from 30 days to 12 months, and operational ingest cost dropped roughly 80%. Atlassian's Data + AI Summit talk puts the daily scale at many petabytes of security data. Winner of Databricks' 2026 Transformation Award.

The pipeline

Sources
Endpoint · network · identity · cloud · app
Full-breadth security telemetry
→
Bronze
Raw Delta retention
Source-fidelity, no lossy pre-processing
→
Silver
OCSF normalization
Every source mapped to OCSF; entity resolution
→
Gold
Analyst surface
Detections and investigation
→
Govern
Unity Catalog
Governance and access plane

What composes, what’s brittle

21B+ events, sub-minute. Queryable in under a minute on the lakehouse.
17 s → 5 s. Query times with Photon.
30 days → 12 months. Retention extended on cheaper storage.
~80% lower ingest cost. Operational ingest-cost reduction.
OCSF on Silver. One normalized schema the detection surface depends on.
What SDW did. Read and scored the public architecture; Atlassian built and runs it.

Sources: Databricks customer story "Atlassian Modernizes Threat Detection" (databricks.com/customers/atlassian/security-lakehouse, 2026-03-24; Niels Heijmans, Chief Security Architect; David Cross, CISO) · Atlassian Data + AI Summit 2026 Security Lakehouse talk

Public production architecture teardown

Falcon LogScale — brute-force scan architecture

CrowdStrike-owned (acquired Humio 2021) log platform built on the inverse of conventional indexing: ~1 MB time-series index per day, compressed segments on object storage, brute-force scan of compressed data in CPU cache. Kafka as the streaming backbone; mechanical-sympathy engineering — SIMD, cache-friendly layouts, bloom-filter pre-screening.

1 PB/day

Vendor-published infrastructure sizing — 63 nodes total (18 Kafka + 45 Humio processing). Throughput claim 1 GB scanned in 0.0265s (~37.7 GB/sec) is consistent with cache-resident scan, but is vendor-reported and not independently replicated. Treat the architecture as durable; treat the multipliers as the vendor's.

The pipeline

Ingest
Apache Kafka
18 nodes at 1 PB/day; persistent disks (Kafka requirement)
→
Process
Humio nodes
45 nodes; compress + segment + ~1 MB time-series index
→
Store
Bucket storage
S3 / GCS / Azure Blob — compressed segments
→
Filter
Three-stage pre-scan
Time range → metadata → bloom filter
→
Scan
Brute-force decompress + scan
SIMD vectorized; data lives in CPU cache

What composes, what’s brittle

Index overhead. ~1 MB/day time-series index vs. 50–100% overhead on indexed SIEMs.
Why compression wins. L1/L2/L3 cache ~1 ns vs. SSD ~10,000 ns — smaller data fits in cache.
Best fit. High-volume, time-bounded queries: threat hunting, forensics, long retention.
Where indexes still win. Unbounded full-text search; complex multi-table joins; small (< 10 GB/day) datasets.
Composes with. Kafka backbone; can sit alongside ClickHouse/StarRocks in a MOAR Engine layer.
What's brittle. Vendor-reported benchmarks; recursive graph traversal and faceted full-text not the design center.

Sources: CrowdStrike Engineering, "How Humio Leverages Kafka and Brute-force Search to Get Blazing-fast Search Results" (vendor blog) · Kresten Krab Thorup (Humio CTO), QCon 2019 · CLP (Yu et al., OSDI 2021) for the compression-over-indexing principle

Public production architecture teardown

Cloudflare — ClickHouse + DataFusion on R2

Edge-network analytics at quadrillion-row scale, run on ClickHouse for nearly a decade — and now paired with R2 SQL, a distributed query engine built on Apache DataFusion that reads Parquet directly from R2 object storage via the Iceberg-backed R2 Data Catalog. Two engines, two retention envelopes, one telemetry plane.

1.61 Q events

Queried in under 2 seconds, across a full day. A separate query scanned 96 trillion events in one hour at the same latency. Margin of error under 1%. Latency held during a simulated North America data-center outage — the active-active soft-cluster design redistributes load without consensus overhead.

The pipeline

Sources
300+ edge data centers
DNS · WAF · bot management · security logs
→
Ingest
Pipelines + R2 streaming
SQL-transformed in flight; writes Parquet to R2 as Iceberg
→
Hot store
ClickHouse soft clusters
Active-active; dynamic node assembly; minimal coordination
→
Cold store
R2 + R2 Data Catalog
Parquet on object storage; Iceberg metadata; partition + column stats
→
Query
R2 SQL (Apache DataFusion)
Coordinator + workers; range reads; filter pushdown

What composes, what’s brittle

~10-year run. ClickHouse in production at Cloudflare since the mid-2010s — among the earliest large-scale adopters.
Active-active by design. No Raft/Paxos in the hot path; nodes can be addressed individually or as a soft cluster.
Why DataFusion on R2. Columnar Parquet + range reads + Iceberg pruning means cold-tier SQL without egress to a separate warehouse.
Composes with. Iceberg R2 Data Catalog is engine-agnostic — Databricks and ClickHouse can read the same tables.
What's distinctive. Two query engines on one storage plane, each chosen for the latency envelope it actually wins on.
What's brittle. Soft-cluster operational discipline; R2 SQL is new (2025) — engine maturity vs. the decade of ClickHouse battle-testing.

Sources: ClickHouse engineering blog, "Trouble will find you: How Cloudflare uses ClickHouse to scale analytics at quadrillion-row scale" (2025) · Cloudflare engineering blog, "R2 SQL: a deep dive into our new distributed query engine" · Cloudflare Pipelines product documentation (SQL transformations + Iceberg, Sep 2025).

Public production architecture teardown

Comcast — Security data fabric on Snowflake

Cybersecurity-at-Comcast moved off siloed, single-tool analytics onto a unified Snowflake-backed security data fabric. Schema normalization across endpoint, identity, cloud, and network telemetry; elastic compute for IOC sweeps, threat-hunting deep dives, and continuous controls assurance from the same store. Later commercialized as Comcast Technology Solutions' DataBee.

10+ PB

Security telemetry under hot retention longer than one year. Automated sweeps of 50,000+ indicators of compromise across the full 10 PB complete in under 30 minutes — a query envelope that's structurally out of reach on indexed SIEM hot tiers at the same retention cost.

The pipeline

Sources
Endpoint · identity · cloud · network
Enterprise-wide telemetry across business units
→
Normalize
Unified schema
Single semantic layer across previously siloed tools
→
Store
Snowflake (separated storage + compute)
10+ PB; >1-year hot retention; elastic warehouse sizing
→
Detect
ML models + IOC sweeps
50K IOCs across 10 PB in <30 min; timeseries analytics for threat detection
→
Serve
SOC + compliance + executive
Self-service dashboards; continuous controls assurance; risk-posture views

What composes, what’s brittle

50K IOCs / <30 min. Sweep across the full 10 PB envelope — the elastic-compute claim made concrete.
Why Snowflake. Separated storage and compute lets IOC sweeps, ML training, and ad-hoc hunts run without contending for one fixed cluster.
Schema-on-write. Unified semantic layer normalizes at ingest — queries don't repay the parsing tax that schema-on-read SIEMs charge per dashboard.
Productized. Now Comcast Technology Solutions' DataBee — security/risk/compliance data fabric for large enterprises.
What's distinctive. Same store serves SOC analysts, compliance, threat hunters, and executives without copy-out.
What's brittle. Snowflake credit consumption at high-frequency dashboard queries; warehouse-sizing discipline matters; the 2024 Snowflake-tenant credential incidents underline customer-side MFA hygiene as a control, not a vendor flaw.

Sources: Snowflake customer case study, "Cybersecurity at Comcast Integrates Snowflake Into Its Security Data Fabric" · Comcast Technology Solutions DataBee press release · SDxCentral coverage ("Comcast's security data fabric taps Snowflake Data Cloud for scalability").

Public production architecture teardown

DNB — DuckDB + Ibis + marimo, off Databricks

Norway's largest financial services group moved its Cyber Defense Center off Databricks notebooks onto an in-house platform built from composable open-source components. Threat-detection workflows wanted single-node compute on high-memory machines; the distributed-by-default Databricks pricing model was a structural mismatch. Ibis as the query layer; DuckDB, Splunk, and Snowflake as interchangeable backends; marimo as the analyst notebook.

Off Databricks

DNB Staff Engineer Kyrre Wahl Kongsgård's first-person account — single-node analyst workloads on millions of security events daily (EDR logs, cloud audit trails, network connection records, process execution tracking) run on DuckDB and Ibis. Splunk remains an addressable backend; Iceberg on Azure Blob holds the long tail.

The pipeline

Sources
EDR · cloud audit · network · process
Millions of events/day across banking, insurance, asset management
→
Store
Iceberg on Azure Blob
Long-retention table format; Snowflake-compatible after the Delta migration
→
Query
Ibis (DuckDB / Spark / Snowflake)
Write once, execute across backends
→
Interface
marimo notebooks
Reactive execution; native Ibis; SPL passthrough to Splunk
→
Specialized
Neo4j · Theseus · Graphistry
Graph analysis and GPU-accelerated investigations on demand

What composes, what’s brittle

Why leave Databricks. Single-node, high-memory threat-detection work paid the distributed-cluster pricing tax for capacity it never used.
Why DuckDB. Embedded OLAP; full SQL; no cluster; fast enough on the working set that single-node beats distributed for the analyst inner loop.
Why Ibis. Decouples the query expression from the engine — analysts can target DuckDB, Splunk, or Snowflake from one notebook.
Why marimo. Reactive notebooks with native Ibis and anywidget; embedded EDR process trees and context-aware investigation state.
What's distinctive. A regulated bank's named CDC engineer publishing the move-off first-hand, so nothing here is reverse-engineered from a vendor case study.
What's brittle. Multi-backend query translation (SPL ↔ Ibis); coordination across specialized engines; provisioning discipline for on-demand analyst notebooks.

Sources: marimo case study, "Why DNB's Cyber Defense Center team moved off Databricks notebooks and onto marimo" — authored by Kyrre Wahl Kongsgård, Staff Engineer at DNB · DNB CDC entry on FIRST.org · Ibis project documentation.

Public production architecture teardown

Pinterest — Zero Trust FGAC at Trino + Gravitino

Pinterest's Monarch big-data platform — 30+ Hadoop YARN clusters, 17k+ nodes on AWS EC2, petabytes processed daily — runs Trino as one of several engines on top of a Credential Vending Service that issues per-user STS tokens scoped by IAM Managed Policies. Apache Gravitino sits as the Iceberg REST catalog; production-validated and named by Pinterest's Director of Big Data Platform on the ASF TLP graduation announcement (June 2025).

17k+ nodes

Monarch scale: 30+ YARN clusters processing petabytes and hundreds of thousands of jobs daily on EC2. CVS dynamically assembles AWS STS tokens at job-launch time — the base IAM role is never returned without at least one modifying Managed Policy attached. Zero standing privilege at the storage tier.

The pipeline

Identity
mTLS · OAuth · Kerberos
User authn; LDAP group membership drives policy selection
→
Broker
Credential Vending Service
AssumeRole with session Managed Policies; intersection-scoped permissions
→
Catalog
Apache Gravitino (Iceberg REST)
First open-source Iceberg REST catalog; production at Pinterest pre-TLP
→
Engine
Trino + Spark + Flink
Multi-engine query layer; Querybook + Jupyter for analyst surfaces
→
Store
S3 (Iceberg + raw)
Hadoop S3A custom credentials provider presents user-specific tokens

What composes, what’s brittle

Per-job STS tokens. CVS issues short-lived credentials; permissions intersect Managed Policies with the base role.
Why Trino here. ANSI SQL across federated lake + catalog without copying data; analyst access is the dominant query pattern.
Why Gravitino. Open Iceberg REST catalog; ASF TLP June 2025; Pinterest named in the graduation announcement as a production user.
What composes. Same CVS-vended credentials flow through Spark, Flink, Hive, and Trino — one trust model, many engines.
What's distinctive. Zero standing privilege at the S3 layer for an analyst population at petabyte scale, running in production.
What's brittle. Kerberos host/service-name discipline; CVS single-instance failure modes; LDAP group sync latency on permission changes; cache-expiry windows of minutes before new policies take effect.

Sources: Pinterest Engineering, "Securely Scaling Big Data Access Controls At Pinterest" (engineering blog, primary source) · Apache Gravitino TLP graduation announcement (June 2025) — quote from Ang Zhang, Director of Big Data Platform, Pinterest · Apache Software Foundation press release.

Public production architecture teardown

Ziggiz — Cyber Lakehouse-as-a-Service on Databricks

First public production reference to ship the Databricks-native Cyber Lakehouse pattern as a service — Delta Lake for storage, Unity Catalog for governance, Spark for batch normalization, separated storage and compute. Tenant onboarding compressed from a 9-month enterprise build to a 5-day managed deployment. DARPA-funded for entity enrichment; selected by Palantir's startup fellowship for asset management.

9 mo → 5 days

Tenant onboarding for new Cyber Lakehouse customers — documented in Ziggiz's A.Team case. Cost savings claim is 30-50% versus three leading SIEMs (per Ziggiz's own press materials), not the higher 90% figure that circulates in derivative coverage. Treat the onboarding number as the claim the case rests on.

The pipeline

Sources
Security telemetry feeds
EDR · cloud · identity · network — tenant-specific connectors
→
Normalize
Apache Spark on Databricks
Schema-on-write; entity enrichment (DARPA-funded line)
→
Store
Delta Lake
Time travel; ACID writes; columnar Parquet under the hood
→
Govern
Unity Catalog
Tenant isolation; lineage; access policy at the table layer
→
Serve
SOC + analyst surfaces
Cyber Lakehouse query plane; Alpha Level integration for alert refinement

What composes, what’s brittle

9 mo → 5 days. Onboarding shrunk by ~50× in the published A.Team case.
30-50% cost. Ziggiz's own claim versus three leading SIEMs — narrower than the 90% figure that circulates downstream.
Why Databricks. Delta + Unity + separated compute lets a small vendor offer multi-tenant lakehouse security as a service without rebuilding the platform.
DARPA validation. Independent technical review funding for entity enrichment — rare third-party signal in the cyber-lakehouse vendor field.
Composes with. Alpha Level Alert Refinery for SOC alert fatigue; Palantir startup fellowship for physical-asset management overlay.
What's brittle. Single-vendor lakehouse coupling (Databricks lock-in); the cost claim narrows under scrutiny — Ziggiz publishes 30-50%, not 90%; small-vendor maturity vs. Splunk/Sentinel install base.

Sources: Ziggiz press release, "DARPA validates what security teams always suspected" · Ziggiz + Alpha Level partnership announcement · Databricks "Data Intelligence for Cybersecurity" launch (March 2026, names Ziggiz as a partner).

Public production architecture teardown

Standard Chartered — self-managed SIEM on Databricks

A global systemically-important bank replaced its traditional SIEM with a self-managed security lakehouse on Databricks — a distributed, multi-cloud Delta Lake architecture where detection content runs as code against the bank's own data plane rather than inside a per-GB SIEM. Presented by the Standard Chartered security team at Databricks Data + AI Summit 2025.

80%

Reduction in time-to-detect after the SIEM-to-lakehouse move, alongside 92% faster threat investigation, 60% better detection accuracy, and ~35% cost reduction. These are vendor-published customer figures (Databricks DAIS 2025 / customer use-case set) — the bank is named and on the record, but the multipliers are the bank's and Databricks' own, not independently reproduced.

The pipeline

Sources
Multi-cloud security telemetry
Endpoint · identity · cloud · network across a distributed multi-cloud environment
→
Store
Delta Lake (distributed, multi-cloud)
Open table format; long retention without per-GB SIEM economics
→
Govern
Unity Catalog
Access control + lineage over regulated security data
→
Detect
Detection-as-code on Databricks
SQL / PySpark detections against the bank's own data plane
→
Serve
SOC + investigation
Faster triage; 92% faster investigation reported

What composes, what’s brittle

80% faster TTD. Time-to-detect cut after replacing the traditional SIEM.
92% / 60%. Faster investigation and better detection accuracy (bank-reported).
~35% cost. Reduction vs. the prior SIEM model — security telemetry off per-GB licensing.
Why self-managed. Distributed multi-cloud lakehouse keeps detection on the bank's own data plane instead of a vendor's query tier.
Why it's credible. Presented by Standard Chartered's own team at DAIS 2025 — a named global bank, on the record.
What's vendor-reported. Metrics are Databricks/SCB-published, not independently replicated; treat the architecture as durable, the multipliers as the bank's.

Sources: Databricks Data + AI Summit 2025 session "Revolutionizing Cybersecurity: SCB's Journey to a Self-Managed SIEM" · Databricks "Data Intelligence in Action: 100+ Data and AI Use Cases" (Standard Chartered security use case).

Public production architecture teardown

Bank Hapoalim — federated lakehouse on Trino/Starburst

Israel's largest bank migrated off Hive onto Starburst (Trino) over a Hadoop-based data lake — federated SQL access that leaves data where it lives. The published case is analytics-and-cost-led; its security relevance is the governance layer: granular federated access controls for cross-border data-residency rules, and ML models for near-real-time AML monitoring built on the federated data.

Federated

Access without centralization — Trino's ANSI SQL over disparate sources, with the granular controls a regulated bank needs to enforce international data rules. The bank's own quote: "ETL processes that took many months at high cost have become extremely fast and accessible to analysts at negligible cost." No security metrics are published; this is a regulated-industry data-federation pattern, not a SOC-telemetry deployment.

The pipeline

Sources
Disparate regulated data
Legacy systems + data lake; cross-border / data-residency constraints
→
Store
Hadoop data lake
Data stays where it lives; no forced centralization
→
Query
Starburst (Trino)
Federated ANSI SQL; SAS / Qlik integration for analysts
→
Govern
Granular access controls
Enforce international data-residency rules at query time
→
Apply
Near-real-time AML monitoring
ML models on the federated data for anti-money-laundering

What composes, what’s brittle

Why federation. Cross-border data-residency rules make leave-it-in-place access the security-relevant choice.
Security angle. Access governance + AML monitoring — compliance / financial-crime, not threat-detection telemetry.
What's published. Analytics and cost outcomes; no security metrics are disclosed in the case.
Why it's here. A named regulated-finance federated-lakehouse pattern — the access-governance lesson generalizes.
Scope. Thinner on security than the other teardowns; included as a governance / data-residency reference, not a SOC build.
Engine. Trino via Starburst — ANSI SQL keeps query logic portable.

Sources: Starburst case study, "Bank Hapoalim" (starburst.io) · Starburst / Trino federated-access and AML-monitoring coverage.

Public production architecture teardown

Yale New Haven Health — SIEM modernization with Cribl + Sentinel

A major US health system hit its Splunk license ceiling when a Palo Alto software update added 63 fields to every firewall log, pushing daily ingest from 400 GB to 600–700 GB. Working with the partner Security Risk Advisors, the security team moved to Microsoft Sentinel as the SIEM and Azure Data Explorer as the data lake, with Cribl Stream already sitting in front as the pipeline tier. The healthcare-specific lesson is that the pipeline layer, not the SIEM, is where a regulated provider regains control of log economics.

40%

Reduction in Palo Alto firewall log volume — enough to bring daily ingest back under the 400 GB/day license, down from 600–700 GB — with a reported 40% cut in SIEM spend. The new SIEM and data lake stood up in about two weeks because Cribl Stream was already in place; leaving Splunk meant repointing outputs. These are vendor-published customer figures (Cribl customer story / Microsoft Ignite 2025), named and on the record, not independently reproduced.

The pipeline

Sources
~30,000 endpoints
Hospital endpoints, Palo Alto firewalls, and diverse healthcare log sources
→
Route
Cribl Stream
Collection, normalization, and filtering; forwards only high-value logs
→
SIEM
Microsoft Sentinel
Detection + analyst tier; replaced Splunk in ~2 weeks by repointing Cribl outputs
→
Lake
Azure Data Explorer (ADX)
Lower-cost retention tier for full-fidelity logs alongside Sentinel
→
Serve
Health-system SOC
Detection + investigation; migration run with partner Security Risk Advisors

What composes, what’s brittle

40% log cut. Palo Alto firewall volume reduced by forwarding only high-value, filtered logs — back under the 400 GB/day cap from 600–700.
What triggered it. A Palo Alto update added 63 fields per firewall log; a 30–45% volume jump blew the Splunk license. The trigger was schema bloat, not more traffic.
Two-week cutover. Cribl Stream already sat in front of Splunk, so switching SIEMs meant repointing outputs — Sentinel + ADX came up in ~2 weeks.
Pipeline is the control point. Log-cost control was regained at the pipeline tier (Cribl), not by changing SIEM vendors — the SIEM swap rode on top of it.
Healthcare relevance. A named US health system across ~30,000 endpoints — the only public healthcare SOC-telemetry case in this catalog.
What's vendor-reported. Figures are Cribl/Microsoft-published with the customer named; treat the architecture as durable, the 40% multipliers as the vendor's and customer's own.

Sources: Cribl customer story, "Yale New Haven Health Reduced SIEM Spend by 40% with Cribl" (cribl.io/customers/yale-new-haven-health) · Microsoft Ignite 2025 session THRSP885, "How Yale New Haven Health modernized security with Sentinel and Cribl" · Security Risk Advisors (SRA), migration partner.

See the full teardowns page →

Dremio — semantic layer + Reflections Trino — federation breadth RisingWave — streaming threat detection Cribl Search — query in place, never rehydrate StarRocks + ClickHouse — engine specialization Apache Spark — security data architecture

Component reference

Dremio — semantic layer + Reflections

Engine-anchored architecture for organizations with mixed BI and security analytics on shared Iceberg data. Reflections pre-materialize the hot paths; the semantic layer keeps the SQL surface clean for analyst and engineer alike.

Iceberg-native semantic-layer engine for organizations with mixed BI and security analytics on shared data. Reflections pre-materialize the hot query paths transparently, and the semantic layer keeps the SQL surface clean for analyst and engineer alike — engineers tune the accelerations, analysts just benefit.

The pipeline

Sources
Security telemetry
Network, endpoint, cloud, identity
→
Route
Vector / Cribl / Kafka
OCSF normalization on ingress
→
Store
Iceberg on S3
Polaris / Nessie catalog
→
Engine
Dremio + Reflections
Semantic layer; materialized accelerations
→
Serve
BI + SOC UIs
Grafana · Superset · notebooks

What composes, what’s brittle

Iceberg-native. Queries hit the lake without copy-out.
Reflections. Accelerate hot queries transparently; engineers tune, analysts benefit.
Semantic layer. Reduces SQL complexity for SOC analysts.
Best fit. Mixed BI + security analytics on shared data; reusable accelerations.
Trade-off. The acceleration story is the Reflections layer: without it, the engine leans on raw scan speed, so Dremio earns its place for the semantic layer and reusable materializations rather than for cold-scan latency.
What survives. Standard SQL detections portable to Trino, StarRocks, ClickHouse-Iceberg.

Sources: Methodology: published spec, reproducible on equivalent hardware · Dremio engineering documentation

Component reference

StarRocks + ClickHouse — engine specialization

Two engines reading the same Iceberg data, each pinned to the workload it measurably wins on. ClickHouse for scheduled queries (dashboards, alerts, sub-second materialized views). StarRocks for ad-hoc investigations (cost-based optimizer, multi-table joins, flexible schema). The SDW join bench (single host, Tier B) confirms the role assignments point the right way, though the gaps are small at SOC scale, so the pairing is justified by isolation, right-sizing, and operating cost more than by raw speed. The dual-engine pattern is MOAR Principle #5 — query engine specialization.

$3.04K/mo

Two right-sized clusters (4× r5.2xlarge each) at ~10 TB/day enterprise scale. Smaller than a single shared cluster forced to serve both workloads. Modeled on AWS on-demand pricing; reproducible figure, not a measured deployment.

The pipeline

Store
Iceberg on S3
Single source of truth; both engines read the same data
→
Scheduled
ClickHouse cluster
Materialized views; <500 ms dashboard refresh; 100K+ events/sec ingest
→
Ad-hoc
StarRocks cluster
Cost-based optimizer; multi-table joins; flexible schema evolution
→
Notebook
DuckDB (optional)
Embedded analytics on the same Iceberg tables
→
Serve
Grafana + analyst UIs
Dashboards routed to CH; investigations routed to SR

What composes, what’s brittle

Why ClickHouse for scheduled. Incremental materialized views on insert; columnar scans tuned for time-series.
Why StarRocks for ad-hoc. CBO + hash/merge/broadcast join strategies; CTEs and BETWEEN time joins.
Same Iceberg data. No copy-out; 20+ engines support the spec, so detection content stays portable.
Where ClickHouse struggles. Multi-table joins, now lab-measured: StarRocks won 4 of 5 TPC-H-derived joins and ran the heaviest 6-table join 2.7–2.8× faster than both ClickHouse arms, though every SOC-suite join finished under 1.5 s on every engine (single host, Tier B). The vendor TB-scale claim (30–120 s vs. 5–30 s) is a different regime the bench neither confirms nor refutes.
Where StarRocks struggles. Aggregation shapes: ClickHouse took q18, the most aggregation-shaped join (1.186 s vs. 1.893 s), and StarRocks won no single-table shape on the earlier 100M-row workload matrix, where the ClickHouse full-scan count ran 10.5 ms to StarRocks' 48.2 ms (both single host, Tier B). The ecosystem is also smaller than ClickHouse's.
What's brittle. The join specialization is now lab-measured rather than vendor-claimed, but it is small at SOC scale, and the sharpest finding was operational: ClickHouse's own native table was the bench's only DNF (q5, >300 s) while the same engine over Iceberg answered in 1.35 s (single host, Tier B; mechanism instrumented 2026-06-10: stats-blind greedy join reordering on the native path built a ~1.2B-row intermediate, and with reordering disabled the same table answers in 5.3 s). Choose on catalog maturity, concurrency behavior, and operating cost; pilot on real OCSF workload before committing.

Sources: ClickHouse documentation + Netflix/Huntress production references · StarRocks documentation and vendor benchmarks (starrocks.io) — TB-scale claims still without independent validation · SDW engine-join-specialization bench, scored June 2026 (first-party lab measurement, single host, Tier B — a different scale regime) · ClickBench (benchmark.clickhouse.com)

See the full components page →

Splunk Machine Data Lake Databricks Lakewatch — open, agentic SIEM Fivetran + dbt — ELT for the security data lake Security data pipeline platforms — the in-flight tier Dremio + VAST Data — Zero Trust cyber lakehouse

Vendor blueprint · prerelease

Splunk Machine Data Lake

Announced September 8, 2025 at .conf25; alpha confirmed February 2026; no GA date public. Splunk's response to lakehouse-native security: a schema-less, AI-ready landing zone inside Splunk Cloud / Enterprise, plus Borderless Real-Time Search federating across S3, Snowflake (GA July 2026), and (announced, unshipped) Iceberg, Delta Lake, and Azure.

What ships today

Cisco Time Series Foundation Model: 250M params, Apache 2.0 open weights, decoder-only multiresolution, 16.12% MASE improvement on observability data. Trained on 300B+ data points. Runs anywhere via PyTorch. PyPI package cisco-tsm.

What doesn't ship yet

Machine Data Lake itself (alpha, no GA date). Federated Search for S3 re-architected (alpha). Snowflake federation (GA target July 2026). Iceberg, Delta Lake, Azure (announced, unshipped). Borderless Real-Time Search engine architecture undisclosed in public docs.

What it changes for architects

The lakehouse pivot is real but pre-shipped. Query plane still routes through Splunk Cloud / Enterprise. Splunk co-founded OCSF and the Cisco TSM is open-source — ecosystem alignment is genuine. The catalog and query-plane control remain Splunk's.

The critique

Forrester flagged the timing gap publicly: "competing platforms already deliver these offerings." No public pricing meter. No named beta customers (Singapore Airlines is a general Splunk reference, not an MDL reference). Decision today: wait vs. run the benchmark yourself.

Sources: Cisco/Splunk press release (2025-09-08) · Forrester .conf25 recap · Splunk "Complete Guide to Data Management" · arXiv 2511.19841 (Cisco Time Series Model) · GitHub splunk/cisco-time-series-model

Vendor blueprint · ELT pattern

Fivetran + dbt — ELT for the security data lake

Managed extraction (Fivetran) plus in-warehouse transformation (dbt) as the ELT spine of a security data lake: Fivetran lands cloud, identity, and SaaS logs into Snowflake / BigQuery with privacy controls; dbt normalizes raw logs to OCSF inside the warehouse, with no separate transform compute. GA products — the security-specific public evidence is real, but thinner than the general data-engineering reputation suggests.

What ships today

Fivetran: managed connectors (AWS, GitHub, Jira, identity providers) into security data lakes; column blocking, hashing, and RBAC before data lands; Hybrid Deployment keeps pipelines inside the network perimeter; webhook push of Fivetran's own logs to Google Security Operations. dbt: version-controlled ELT mapping disparate logs to OCSF inside Snowflake.

Where the public evidence is

Concrete and security-specific: Heritage Environmental Services (Fivetran Hybrid Deployment for regulated data); the documented Fivetran → Google Security Operations webhook integration; the published dbt → OCSF normalization pattern in Snowflake. The pattern is verifiable.

Where it isn't

"Brex, Coinbase, Rippling use Fivetran/dbt for security" is reputational, not a named public security case. They are cited for advanced data-engineering practice generally; treat the security-specific attribution as unproven until a public case names it.

What it changes for architects

ELT-to-OCSF moves normalization out of the SIEM and into the warehouse the org already runs. The catch is that managed ingestion is a recurring per-connector cost and a data-egress decision, so weigh it against Cribl / Vector routing, and validate on your own sources before assuming Brex-tier polish.

Sources: Fivetran documentation (Hybrid Deployment; Google Security Operations webhook integration; column blocking / hashing / RBAC); dbt + OCSF normalization public references (dbt Labs / Snowflake security data lake guides).

Vendor blueprint · SDPP category

Security data pipeline platforms — the in-flight tier

The pipeline tier has two shapes. Warehouse ELT (Fivetran + dbt) extracts, loads, then transforms in the warehouse. SDPP route, reduce, reshape, and normalize telemetry in flight — before it lands — sitting between sources and the lake/SIEM. This is where volume economics and OCSF-on-ingress actually happen. The category is crowded and mostly vendor-claimed for security specifics; the public evidence concentrates in one or two players.

What the category is

An in-flight tier between sources and the store: route, reduce (drop / sample / aggregate), reshape, mask PII, normalize to OCSF before landing. Players: Cribl, Vector (CNCF, Datadog-stewarded), Datadog Observability Pipelines, Tenzir, DataBahn, Observo AI, Monad, Abstract Security. Distinct from warehouse ELT — pre-landing, not in-warehouse transform.

Where the public evidence is

Concentrated, not broad. Cribl has named public security cases (Yale New Haven Health, 30K endpoints) — it carries its own component reference in this catalog. Vector is production-validated inside the Huntress teardown on this site (the routing tier at 200K rec/sec). Those two are real and independently citable.

Where it isn't

Datadog Observability Pipelines, Tenzir, DataBahn, Observo AI, Monad, and Abstract Security are credible and largely GA, but their security-specific production validation is vendor-claimed or analyst-relayed, not a named public security case. Treat the newer entrants as unproven for security until a public case names them — the same discipline applied to Fivetran/dbt.

What it changes for architects

The SDPP tier is where 30–70% volume reduction and OCSF-on-ingress land, ahead of SIEM/lake economics. The catch is a fast-consolidating category with heavy vendor claims and thin independent security validation, so choose on measured reduction against your telemetry and on pipeline-config portability over logo count. Cribl and Vector are the proven anchors; the rest are bets.

Sources: Cribl engineering blog; Yale New Haven Health published case (see the Cribl Search component reference); Vector (CNCF / Datadog) production-validated in the Huntress teardown on this site; Datadog Observability Pipelines product documentation; Tenzir / DataBahn / Observo AI / Monad / Abstract Security vendor materials and analyst commentary — security-specific production validation vendor-claimed pending a named public case.

See the full vendor blueprints page →

DetectFlow — detections without operational debt DuckLake — SQL metadata for the lakehouse MLOps for threat hunting Splunk federated integration with the MOAR stack

Methodology

DetectFlow — detections without operational debt

Detection-as-code architecture pattern for SOCs whose detection backlog grows faster than the team can maintain. Versioned, tested, deployable rules; CI/CD pipelines with regression tests; continuous performance and false-positive measurement. The discipline that lets a SOC scale past the hundreds-of-detections operational ceiling.

1000+

Detections in production with operational debt held flat. Most detection programs cap at low hundreds because each new rule adds maintenance load; DetectFlow keeps the per-rule maintenance cost near zero through automation.

The pipeline

Author
Detection content
Versioned in Git; YAML, SPL, KQL, SQL per target engine.
→
Test
CI/CD pipeline
Unit tests, regression suites, replay against synthetic and historical telemetry.
→
Deploy
Detection engine
Splunk ES, Sentinel, Chronicle, custom — engine-agnostic.
→
Measure
Feedback loop
Performance metrics, false-positive rate, MTTD — fed back to backlog.

What composes, what’s brittle

Why this matters. The hundreds-of-detections operational ceiling is real; DetectFlow removes it.
Reference patterns. Anvilogic, Panther, hand-rolled CI/CD on Detection-as-Code repos.
Detection portability. Standard SQL detections move between engines without rewrite.
Feedback loop. Every rule's performance and FP rate measured per deployment cycle.
Best fit. SOCs where analyst time is consumed by tuning rather than hunting.
What's hard. The cultural shift from detection content as personal craft to detection as production software.

Sources: Reference patterns: Anvilogic, Panther, Splunk Enterprise Security content packs; published Detection-as-Code repos including SigmaHQ.

Methodology

MLOps for threat hunting

Operationalizing Model-Assisted Threat Hunting (M-ATH) from the PEAK framework. A notebook model degrades the moment telemetry shifts or an adversary adapts; MLOps is the engineering discipline — feature store, model registry, orchestrated retraining, drift monitoring — that keeps an algorithmic hunt viable in production.

Level 2

CI/CD-automated retraining is the threshold where M-ATH stops being a one-off notebook hunt. Google Cloud's MLOps maturity model: Level 0 (manual notebook, no drift defense) → Level 2 (pipelines that detect drift and retrain autonomously). Below Level 1, the model is stale before the hunt ends.

The pipeline

Prepare
PEAK + feature store
Frame the hypothesis; engineer features once in Feast — consistent offline training and online inference.
→
Train
M-ATH model
Supervised classification, clustering, time-series, NLP on petabyte telemetry; experiments tracked in MLflow.
→
Deploy
Orchestrated retraining
Kubeflow / ClearML pipelines retrain and redeploy on schedule or on drift, as continuous delivery.
→
Monitor
Drift + MLSecOps
Data- and concept-drift detection (W&B); poisoning / evasion guardrails feed back to the retrain loop.

What composes, what’s brittle

PEAK framework. Bianco, Fetterman, Marrone (Splunk SURGe). M-ATH is the algorithmic hunt type alongside hypothesis-driven and baseline.
Data vs concept drift. Distribution shift vs adversary adaptation — both degrade silently into a false-positive avalanche or a false-negative blind spot.
Why Level 0 fails. A notebook model trained offline and discarded cannot counter non-stationary, adversarial telemetry.
Tooling. Feast (features), MLflow (registry / Detection-as-Code), Kubeflow & ClearML (orchestration), W&B (drift + reasoning observability).
MLSecOps. The hunt infra is itself an attack surface — retrain-loop poisoning, evasion inputs, MLflow CVE-2026-2635, the Kubeflow Doki incident. MITRE ATLAS.
What's hard. Continuous-retraining cost; poisoned retraining loops; the org gap between data science and detection engineering.

Sources: Splunk PEAK Threat Hunting Framework (Bianco, Fetterman, Marrone — SURGe); "The Threat Hunter's Cookbook" (Fetterman & Marrone); Google Cloud MLOps maturity model (CI/CD for ML); MITRE ATLAS; CVE-2026-2635 (MLflow authentication bypass); PROID compromise-assessment framework (peer-reviewed, PMC).

Methodology

Splunk federated integration with the MOAR stack

Integration pattern for organizations with deep SPL investment that need lakehouse economics on 30+ day data. Keep Splunk as the analyst UI; route high-value signals to Splunk indexers (30-day hot) and all data to Iceberg (1–3 year cold); bridge them with DB Connect (production, on-prem + cloud) or Federated Search (Splunk Cloud only, beta). Analysts keep SPL; storage cost goes to lakehouse rates.

3.6–10.1×

Speedup vs. the schema-on-read SIEM baseline (OpenSearch 2.18.0, 2.854 s) on the canonical sdw-lab zeek-flagship-rerun (10M Zeek events, single host, Tier B): Trino 3.6× (0.795 s), StarRocks 8.3× (0.343 s), ClickHouse-on-Iceberg 10.1× (0.282 s); ClickHouse native's single-query best case (the old "145×" headline) is superseded by the CV-gated OpenSearch re-run, which puts ClickHouse-native at 46.8× on the five-query average (5–62× on the hunting-shaped queries across the native and Iceberg realizations, with the index winning the simple lookups). Methodology and code public; the schema-on-read tier degrades roughly 8× from 1M → 10M events. The win is the tier move, so pick the cold-tier engine on catalog maturity, concurrency, and operating cost rather than on the spread between those multipliers.

The pipeline

Route
Cribl / Tenzir / Vector
Dual-write: high-value to Splunk, all data to Iceberg
→
Hot tier
Splunk indexers (30 days)
Real-time alerts; native SPL speed for recent data
→
Cold tier
Iceberg on S3 (1–3 years)
OCSF-normalized; columnar Parquet; partition pruning
→
Bridge
DB Connect (JDBC) or Federated Search
SPL query plane spans both tiers
→
Query
Trino / Dremio / StarRocks / ClickHouse
Standard SQL on the cold tier; detection content portable

What composes, what’s brittle

Why this works. Splunk's schema-on-read parsing tax shows up on repeated dashboards; pre-normalize to OCSF and pay once.
DB Connect. Production-grade, on-prem + cloud; 2–5× slower than the schema-on-read SIEM's native search (single host, Tier B), 10–50× cheaper storage.
Federated Search. Splunk Cloud only, beta; direct Iceberg; DSU "use-it-or-lose-it" meter — high-frequency queries can cost more than ingestion.
SPL feature gaps. transaction, datamodel, inputlookup, real-time alerting on federated sources are not supported.
Best fit. Heavy SPL investment + long-retention compliance + analyst retraining cost too high to swallow at once.
What's brittle. Federated Search beta SLA; SPL → SQL translation gaps; multi-cloud catalog independence (Glue vs. Polaris).

Sources: SDW Splunk DB Connect benchmark, December 2025 (github.com/flying-coyote/splunk-db-connect-benchmark) · Splunk Federated Search for S3 documentation · Cisco Data Fabric announcement (Sep 2025) · Splunk .conf25 federation evolution

See the full methodologies page →

Featured teardown · analyzed by SDW

Atlassian’s Project Banyan on Databricks + OCSF

A close case study of Atlassian’s Project Banyan: a lakehouse-native security data platform on the Databricks medallion architecture, with security telemetry normalized to OCSF on the Silver layer, Unity Catalog governance, more than 21 billion events queryable in under a minute. Atlassian built and operates it; this page analyzes the architecture, drawn from their public Databricks story. Internal numbers beyond the published record stay Atlassian’s.

Read the teardown →

How the catalog grows

What it takes to get in

A new entry goes in only when the source material is publicly attributable (a conference talk, an engineering blog with measured outcomes, a vendor case study with the customer named) and when the architecture is mature enough that its failure modes show up in the public record as well as the marketing one. That gate is why I trust a teardown, and why a vendor blueprint stays labeled as unproven until someone names a deployment.

The catalog is free to use, and so is the Matrix, which has been public since July 2026; the engagement work that applied both to a client's own environment is paused while I work full-time.

Working patterns also get contributed upstream to the CISA Joint Cyber Defense Collaborative, so the patterns end up as public knowledge that no vendor owns.

See how the pattern lands on your workload.

The matrix scoring that justified each reference architecture's tool choices is public, and so is the benchmark behind it, so you can reproduce it on your own workload and weight the scores to your own environment.

Read the matrix →See the lab

What’s actually in production.

Named organizations already running it in production.

Today: pick two.Need: focused, evidence-based advocacy on every one.

Atlassian — Project Banyan

Huntress on ClickHouse

Okta on DuckDB-in-Lambda

Microsoft Sentinel on Azure

Google Chronicle on BigQuery

Falcon LogScale — brute-force scan architecture

Cloudflare — ClickHouse + DataFusion on R2

Comcast — Security data fabric on Snowflake

DNB — DuckDB + Ibis + marimo, off Databricks

Pinterest — Zero Trust FGAC at Trino + Gravitino

Ziggiz — Cyber Lakehouse-as-a-Service on Databricks

Standard Chartered — self-managed SIEM on Databricks

Bank Hapoalim — federated lakehouse on Trino/Starburst

Yale New Haven Health — SIEM modernization with Cribl + Sentinel

Dremio — semantic layer + Reflections

Trino — federation breadth

RisingWave — streaming threat detection

Cribl Search — query in place, never rehydrate

StarRocks + ClickHouse — engine specialization

Apache Spark — security data architecture

Splunk Machine Data Lake

What ships today

What doesn't ship yet

What it changes for architects

The critique

Databricks Lakewatch — open, agentic SIEM

What ships today

What doesn't ship yet

What it changes for architects

The critique

Fivetran + dbt — ELT for the security data lake

What ships today

Where the public evidence is

Where it isn't

What it changes for architects

Security data pipeline platforms — the in-flight tier

What the category is

Where the public evidence is

Where it isn't

What it changes for architects

Dremio + VAST Data — Zero Trust cyber lakehouse

What ships today

What isn't shown

What it changes for architects

The critique

DetectFlow — detections without operational debt

DuckLake — SQL metadata for the lakehouse

MLOps for threat hunting

Splunk federated integration with the MOAR stack

Atlassian’s Project Banyan on Databricks + OCSF

What it takes to get in

See how the pattern lands on your workload.