Public production architecture teardown
DNB — DuckDB + Ibis + marimo, off Databricks
Norway's largest financial services group moved its Cyber Defense Center off Databricks notebooks onto an in-house platform built from composable open-source components. Threat-detection workflows wanted single-node compute on high-memory machines; the distributed-by-default Databricks pricing model was a structural mismatch. Ibis as the query layer; DuckDB, Splunk, and Snowflake as interchangeable backends; marimo as the analyst notebook.
DNB Staff Engineer Kyrre Wahl Kongsgård's first-person account — single-node analyst workloads on millions of security events daily (EDR logs, cloud audit trails, network connection records, process execution tracking) run on DuckDB and Ibis. Splunk remains an addressable backend; Iceberg on Azure Blob holds the long tail.
The pipeline
-
Sources
EDR · cloud audit · network · process
Millions of events/day across banking, insurance, asset management
-
Store
Iceberg on Azure Blob
Long-retention table format; Snowflake-compatible after the Delta migration
-
Query
Ibis (DuckDB / Spark / Snowflake)
Write once, execute across backends
-
Interface
marimo notebooks
Reactive execution; native Ibis; SPL passthrough to Splunk
-
Specialized
Neo4j · Theseus · Graphistry
Graph analysis and GPU-accelerated investigations on demand
What composes, what’s brittle
- Why leave Databricks. Single-node, high-memory threat-detection work paid the distributed-cluster pricing tax for capacity it never used.
- Why DuckDB. Embedded OLAP; full SQL; no cluster; fast enough on the working set that single-node beats distributed for the analyst inner loop.
- Why Ibis. Decouples the query expression from the engine — analysts can target DuckDB, Splunk, or Snowflake from one notebook.
- Why marimo. Reactive notebooks with native Ibis and anywidget; embedded EDR process trees and context-aware investigation state.
- What's distinctive. A regulated bank's named CDC engineer publicly publishing the move-off — not a vendor case study reverse-engineered.
- What's brittle. Multi-backend query translation (SPL ↔ Ibis); coordination across specialized engines; provisioning discipline for on-demand analyst notebooks.
Sources: marimo case study, "Why DNB's Cyber Defense Center team moved off Databricks notebooks and onto marimo" — authored by Kyrre Wahl Kongsgård, Staff Engineer at DNB · DNB CDC entry on FIRST.org · Ibis project documentation.