Methodology meets architecture

PEAK and the lakehouse: how modern data stacks enable threat hunting.

Threat hunting without a framework is just ad-hoc investigation: analysts following hunches with no systematic approach. The PEAK framework from Splunk's SURGe research team (David Bianco, Ryan Fetterman, Sydney Marrone) gives that work structure: Prepare, Execute, Act with Knowledge. PEAK was written for the SIEM era, and the operational workflow still holds, but the data architecture underneath it has changed.

Reading time: about 16 minutes. Evidence tier: B overall. PEAK is a published Splunk SURGe framework (Tier A for the framework itself), the lakehouse mapping draws on practitioner deployments and reference architectures (Tier B), and the cost comparisons are directional estimates based on vendor pricing and my own TCO models. I flag the speculative pieces inline.

The Prepare phase covers threat-intelligence review, hypothesis selection, scope definition, and verification that the required data sources are actually retained for the window the hunt needs. Traditional SIEM architectures squeeze this phase from one specific direction, retention cost.

Schema-on-read SIEMs typically charge $2–4/GB/month for indexed, searchable storage. At 10 TB/day ingestion, 90-day retention costs roughly $2–3 million per month at list pricing. Extending to a full year approaches $10–12 million per month. Most organizations don't pay that; they archive to cold storage and accept that data older than 90 days is effectively unsearchable without rehydration. That economic reality dictates what hunts are even possible to prepare for.

Mandiant's M-Trends reports have consistently shown median adversary dwell times in the range of 10–20 days, but the 75th-percentile tail extends well past 50 days, and named APT campaigns routinely maintain persistence for 6–18 months before discovery. A 90-day retention window can investigate most commodity intrusions, but it cannot prepare a hunt against a slow-moving adversary that established persistence eight months ago.

A lakehouse architecture changes the storage economics by roughly two orders of magnitude. Iceberg or Delta tables on S3 or ADLS Standard-IA run $0.01–0.02/GB/month for the underlying object storage. The same 10 TB/day at three-year retention costs roughly $13K–25K/month in storage rather than $32M/month in indexed SIEM tier, which isn't a tuning gain so much as a structural change in what Prepare can consider on day one.

The Prepare phase also depends on metadata: knowing what data sources exist, what schema they're on, when coverage started and stopped. Wiki pages and Confluence documents are the default in most SOC environments. They're also chronically out of date. A catalog-backed lakehouse (Polaris, Unity Catalog, Glue) makes that metadata queryable. Before starting a hunt, I can run a SQL query against the catalog to confirm which tables contain the relevant fields, what their earliest timestamp is, and whether schema changes during the proposed hunt window will affect query construction.

I want to flag what doesn't change, because threat-intelligence review, MITRE ATT&CK familiarity, and hypothesis quality are analyst-skill questions, and while the architecture can make more data available, it cannot make the analyst better at picking hunts, which is something PEAK gets at and the lakehouse has nothing to say about.

PEAK's Act phase is the bridge between hunting and detection engineering. A successful hunt (one that produces a confirmed adversary pattern) turns into a deployed detection. The deployment shape used to be: write a SIEM correlation rule, schedule it to run every 5–15 minutes, document the rule in a wiki page, and call it done. That cadence was tolerable when the relevant adversary timeline was measured in hours or days.

The 2026 numbers don't support that cadence anymore. CrowdStrike's 2026 Global Threat Report measured a 27-second fastest recorded adversary breakout time, the interval from initial access to lateral movement. Mandiant's M-Trends 2026 reported negative mean time to exploit (MTTE) for some campaigns, meaning exploits landing days before patches were available. A detection cadence of 5–15 minutes misses the response window for a 27-second breakout by more than an order of magnitude, and that gap is structural rather than a matter of tuning the schedule tighter.

Pipeline-based detection (running detection logic in-stream against the event flow before data lands in storage) is the architectural response. Tools in this space include Tenzir, Cribl Stream, Apache Flink, and the streaming-detection layer inside platforms like Lakewatch. The pattern is consistent: events flow through a processing pipeline that applies detection logic, emits high-confidence findings as a separate signal stream, and writes the underlying telemetry to the lakehouse for hunting and forensics. Detection latency drops from minutes to seconds or sub-second.

For the Act phase specifically, this changes two things. First, the analyst's hunt query becomes deployable as a pipeline operator, not only as a scheduled SIEM rule; many platforms accept Sigma rules or SQL fragments that translate directly into stream operators. Second, the documentation artifact changes shape. Rather than a wiki page that drifts, the detection's metadata (hypothesis, data sources, precision target, validation dataset, MITRE ATT&CK mappings) lives in the catalog as structured data, queryable alongside the hunts that produced it.

I'd put a caveat on the latency claim. Pipeline detection is sub-second to seconds for in-stream rules. For detections that require correlation across a longer window (say, "credential dumping followed within ten minutes by lateral movement") the latency depends on the windowing model and the state store, and is closer to seconds-to-tens-of-seconds. That's still an order of magnitude better than 5–15 minute SIEM scheduling, but it's not zero. Cross-link to pipeline detection and streaming for the deeper technical picture.

M-ATH is PEAK's name for the model-assisted hunt type, and the phrase that captures it best is "human hypothesis, machine breadth." The analyst generates the question, the model assists with narrowing a million-row result set to a few hundred candidates worth human review, and the analyst then validates, confirms true positives, rejects false positives, and uses the validated set to either deploy a detection or refine the model.

The technique used most often in published M-ATH work is unsupervised anomaly detection, where Isolation Forest (Liu et al., 2008) is the canonical choice for tabular security data because it handles high-dimensional feature spaces well and produces an interpretable per-row anomaly score. DBSCAN clustering and autoencoder-based approaches also appear. The common shape: the model surfaces the statistically unusual subset, and the analyst applies domain knowledge to separate "unusual and bad" from "unusual but legitimate."

The lakehouse is well-suited to M-ATH for a specific reason: the same data the model trains on is the same data the production detection scores against. There's no train-test-deploy gap where the model is built on a sampled extract and then deployed against a different schema. The DuckDB-in-notebook pattern is the typical prototyping path: load Arrow buffers from Iceberg, fit a scikit-learn or PyTorch model, validate against held-out data, then either deploy the model into the pipeline (Tenzir, Flink) or translate the model's high-confidence threshold into a SQL detection rule for StarRocks.

I want to be careful about overclaiming what M-ATH delivers. Published precision numbers for Isolation Forest on authentication anomalies vary widely; I've seen anything from 30% to 85% depending on the feature set and the labeling rigor. The technique works best when the analyst has already reduced the search space via Prepare-phase hypothesis selection, and when there's a validated baseline to compare against. Dropped onto raw event data with no priors, anomaly detection produces mostly noise. That's not an M-ATH limitation; it's a property of unsupervised learning on high-cardinality real-world data.

For the operational side of running M-ATH at scale, see MLOps for threat hunting in the reference architectures section, which covers model registry, validation datasets, and the feedback-loop discipline that separates a working M-ATH program from a notebook full of one-off experiments.

PEAK assumes a certain operational maturity level. The framework's value compounds with the organization's place on the Hunting Maturity Model (HMM) ladder, also a David Bianco contribution, predating PEAK. HMM levels run from 0 (no hunting capability) through 4 (leading, automated, feedback-driven hunting program).

For HMM0–1 organizations, PEAK is aspirational. The framework can guide which hunts to attempt first, but the data infrastructure and analyst skill aren't there yet to fully execute. The lakehouse work in this phase is mostly foundational: get telemetry into Iceberg, validate that the catalog reflects reality, run a few baseline queries to confirm the data is usable.

For HMM2–3 organizations, PEAK is operational. Hypothesis-driven and baseline hunts are running on a regular cadence. The lakehouse contribution is removing latency from the Execute phase and giving the Knowledge phase a structured home. Most organizations I see in this band have the analyst skill already; what they don't have is the data platform that lets the analysts work without waiting 10 minutes per query.

For HMM4 organizations, PEAK is just the methodology; the work shifts to M-ATH, automated detection deployment, and feedback loops between hunting and detection engineering. The lakehouse is essential rather than optional; the same data needs to serve exploratory hunts, model training, production detection, and forensic investigation, without architectural duplication.

For the deeper HMM walkthrough and how the maturity progression maps to specific architecture decisions, see detection engineering maturity in this series.

A few places where the "lakehouse enables PEAK" claim deserves more friction than the marketing version of this argument tends to apply:

• Operational maintenance is real cost. Iceberg and Delta tables need compaction, snapshot expiration, and schema-evolution discipline. A neglected lakehouse runs slower than a tuned one and accumulates technical debt that compounds. SOC teams without data-engineering support tend to underestimate this.
• Schema drift kills hunts silently. When an upstream source changes a field name or a vendor renames a sourcetype, downstream hunt queries break in ways that may not surface until an analyst notices a hunt is suspiciously quiet. OCSF helps; LLM-assisted OCSF mapping helps more; neither eliminates the problem.
• The cost comparison is generous to the lakehouse. SIEM list pricing is a high bar. Most organizations negotiate substantial discounts (30–60% is typical). The "$2–3M/month at 10 TB/day" number is list, not actual. The lakehouse advantage is real but probably closer to 5–10× than the 100× the raw math suggests.
• Pipeline-detection skill is scarce. Most SOC analysts can write SPL or KQL. Far fewer can write Tenzir pipelines, Flink jobs, or streaming SQL. The capability change matters; the hiring market is the binding constraint for most teams.
• PEAK works without a lakehouse. The framework is methodology, not architecture. Splunk-shop SOCs run PEAK successfully today. The lakehouse changes the economics and the latency profile; it does not change whether the framework applies.

None of these are reasons to dismiss the architectural shift. They're reasons to size expectations correctly and treat lakehouse adoption as an engineering investment with operational consequences, not a free win.