Detection engineering

Where detection-as-code notebooks should live.

A notebook is how a hunt, an investigation, or a candidate detection gets shared so a second analyst can re-run it six months later. Security teams have done this for years, so the open question isn't whether to use notebooks, it's which notebook to commit the durable work to. There's a real fork here: the Databricks notebook that works in production today but moves the lock-in up to the authoring layer, and the open, reproducible marimo notebook that fixes the things that make Jupyter untrustworthy in production but has no security ecosystem standing behind it yet. Underneath both, the detection that actually ships is text.

Reading time: about 18 minutes. Evidence tier: B–C overall, with the reproducibility mechanics graded higher. The notebook-mechanics claims rest on Pimentel et al. (MSR 2019, Tier A) and the marimo and Jupyter project docs; the Databricks platform claims are vendor marketing and flagged Tier C in line; the "marimo is better for security work" claim is labelled Tier D, because no security team has published it yet.

If you went looking, a few years ago, for security use cases shared as notebooks, you found one cluster and one library. The cluster is Roberto Rodriguez's: the ThreatHunter-Playbook (ATT&CK-organized hunt analytics published as interactive notebooks and a Jupyter Book, around 4,600 stars), Security-Datasets, formerly Mordor (replayable event data that made those notebooks reproducible, around 1,760 stars), HELK (an ELK-plus-Spark-plus-Jupyter hunting stack, around 3,900 stars), and the Infosec Jupyterthon, the online event where researchers presented their favorite notebooks. The library is Microsoft's msticpy, which gives a hunting notebook data providers for Microsoft Sentinel, Defender, Graph, and Splunk, plus enrichment and analysis. (Repo metadata Tier A, pulled from GitHub in June 2026; the maintenance reads below are from commit history.)

Here is the part that surprised me when I went back to check. Most of that corpus is dormant. Security-Datasets has not had a commit since March 2024, HELK since mid-2024, and the Infosec Jupyter Book primer since 2020; the Jupyterthon's last confirmed edition was February 2024. The ThreatHunter-Playbook still shows a 2026 commit, but it's mid-reinvention toward LLM agent tooling rather than steady notebook curation. The one project that's unambiguously healthy is msticpy, last touched the day before I wrote this, and msticpy is a library, not a notebook collection. So the durable thing that survived is the code you import into a notebook, not the notebooks themselves.

I don't read that as "notebooks failed." I read it as notebooks finding their level. They turned out to be very good for exploration and for sharing a worked analysis, and weak as the system of record for production detections, and the engineering that needed to be durable migrated into libraries (msticpy, Google's secops SDK) and into text-based tooling that makes notebooks behave in git (jupytext, nbdime, both actively maintained). That a thriving toolchain exists specifically to paper over notebooks-in-version-control is the tell that the version-control problem is real, and it's the problem the rest of this essay is about.

Databricks' security story runs on notebooks in a way the Jupyter corpus never quite managed, because the notebook there is also the production runtime. The security solution accelerators ship as runnable Databricks notebooks: the DNS-analytics accelerator ("Detecting Criminals and Nation States through DNS Analytics") and the IOC-matching accelerator are GitHub repos you clone into Databricks Repos and run, with the IOC one wiring continuous matching through Delta Live Tables. So a shared, runnable security use case at Databricks is a notebook, and unlike a ThreatHunter-Playbook notebook you run on a laptop, it's already attached to the engine that holds the data and the scheduler that runs it in production.

The platform has leaned into this hard. Over fourteen months the framing went from "augment your SIEM" to a "Data Intelligence for Cybersecurity" agent platform (September 2025) to Lakewatch, announced 24 March 2026 as an "open agentic SIEM," with Anthropic's Claude as the reasoning layer and acquisitions of Antimatter and SiftD.ai alongside (all Tier C, vendor announcements; SiftD.ai's claimed pedigree as Splunk SPL's creator is their claim, not independently verified). Zerobus, generally available 23 February 2026, is a serverless API that writes events straight into Delta tables and skips the message bus. The honest read on adoption is that named customers have presented their own migrations, most usefully Standard Chartered's "self-managed SIEM on Databricks" talk at the 2025 Data and AI Summit (Tier B, a customer speaking for itself), but nearly every quantified claim routes through Databricks marketing and the specific numbers disagree across sources, so I'll cite the direction and not the percentages.

Taken at face value, this is the most complete answer to "where do production detection notebooks live" that anyone is shipping. Real-time co-editing, per-notebook permissions, git integration, a one-click path from notebook to scheduled job, natural-language querying through Genie. If the question were only about capability, Databricks would be the recommendation and the essay would end here. The reason it doesn't is the format the capability is delivered in.

marimo is a reactive Python notebook, and the reactive part is the answer to Grus's complaint. It parses each cell statically into a dataflow graph and runs cells in dependency order rather than click order, so when a cell changes, the cells that depend on its variables re-run, and when you delete a cell, its variables are scrubbed from memory. Hidden state, the thing that makes a Jupyter notebook lie about what produced its output, is gone by construction rather than by discipline. The price is two structural rules (no cycles between cells, no variable defined in more than one cell), which is exactly the friction that trips up someone porting a messy notebook, and exactly the constraint that makes the execution deterministic.

The second thing marimo does is store the notebook as a pure .py file, no JSON envelope and no embedded output blobs, so a one-line change to a detection is a one-line diff a reviewer can actually read. SQL is a first-class cell type that serializes as a normal Python call and runs against DuckDB or against in-memory dataframes, which fits a lakehouse-backed SOC well. And the same file runs three ways: marimo edit to author interactively, marimo run to serve it as a locked-down app, and python notebook.py to run it headless as a scheduled job, with no marimo server in the loop. The "notebook to production job" step that's a rewrite elsewhere is, here, the same file.

# A marimo notebook IS this .py file — git sees readable code.

import marimo
app = marimo.App()

@app.cell
def _():
    import duckdb
    df = duckdb.sql("""
        SELECT src_ip, query, beaconing_score(query) AS score
        FROM read_parquet('s3://security-lake/dns/*.parquet')
        WHERE event_time > '2026-01-01'
    """).df()
    return (df,)

@app.cell
def _(df):
    # Reactive: edit the threshold and this cell — and only this cell's
    # dependents — re-run. No stale state, no out-of-order surprises.
    matches = df[df.score > 0.9]
    return (matches,)

There's one more piece that turns "git-friendly" into "reproducible months later": marimo can inline its dependencies into the file as PEP-723 metadata and run in a uv-managed sandbox, so the notebook is pinned down to its packages. For detection content you need to trust and re-run during an investigation or an audit, that combination, deterministic execution, readable diffs, pinned dependencies, is the property you actually want, and it's the property plain Jupyter has never had without a stack of bolt-ons. Graded on the notebook mechanics alone, this is the better-built tool, and that claim is Tier A.

The way out of the fork is to stop treating the notebook as the place detections live and start treating it as the place detections are derived. The artifact that runs in production, that fires an alert and that an analyst reads during triage, should be text: a Sigma rule, a SQL statement, a small versioned Python module, committed to the same repository as the rest of your detection content and reviewable like any other code. The notebook is the lab record, the document that explains why the rule has an entropy check and a signed-binary filter and an off-hours condition, so that when the rule fires six months from now and someone asks why it exists, the answer is in the notebook rather than in someone's memory. That's the same point I made about Jupyter and MLflow; the notebook doesn't replace the detection rule, it justifies it.

Hold that distinction and the fork loses most of its teeth. On Databricks, the answer is to keep the durable detection in text, a Sigma rule or a SQL file under version control, and treat the .dbc notebook as scratch and derivation rather than the system of record, so that the proprietary authoring layer never becomes the thing you can't leave. On marimo, the .py file is already text and already diffable, which makes the derivation itself a first-class reviewable artifact, and the missing-ecosystem problem shrinks to "can I get the data into a dataframe," which DuckDB over Parquet or Iceberg largely solves. Either way, the test of whether a team has reached real detection-engineering maturity is the same: a new analyst can answer "why does this detection exist" by reading the file that produced it, and the production detection is portable text that doesn't care which notebook wrote it.

This is the applied-bridge move I keep coming back to, taking a discipline the data-engineering world already worked out, reproducible analysis as reviewable, version-controlled, dependency-pinned code, and carrying it into security, where the default is still a query pasted into a console and lost when the tab closes. The data world solved the reproducible-notebook problem; security mostly hasn't adopted the solution, and naming the fork honestly is the first step toward carrying it across.