Technology deep-dive

The detection engineering maturity ladder.

Most active threat hunting programs I encounter operate at HMM2, where they apply procedures somebody else wrote, and the investment argument that gets them to HMM3, where analysts create novel techniques, is the argument I keep watching organizations lose. This essay is the version of that argument I wish I'd had in writing five years ago.

Reading time: about 20 minutes. Evidence tier: A for David Bianco's framework and SANS adoption, B for Expel production metrics and practitioner interviews, with the HMM-distribution figures and upskilling-timeline numbers flagged where they're survey-derived rather than peer-reviewed.

HMM0 — initial

The team reacts to alerts from IDS, antivirus, and the SIEM's correlation rules. Data collection is whatever the alerting tools happen to capture. Hunting, in the active sense, doesn't occur. Bianco does not consider HMM0 organizations to be hunting at all; they're operating a reactive detection program. That's a defensible posture for some businesses, but worth naming so leadership doesn't silently assume the SOC is doing something it isn't.

HMM1 — minimal

The team is still primarily reactive, but it has begun routine data collection beyond the SIEM's retention window. Analysts run IOC lookups against threat-intel feeds and do historical lookback when a new indicator drops. Operationally this often looks like S3 or Azure Blob holding the previous twelve to twenty-four months of logs, with a thin query layer on top. There's no advanced analytics; the capability is "if a known-bad indicator shows up in feeds, we can check whether we've ever seen it."

HMM2 — procedural

The team applies hunting procedures that other people wrote, whether that's MITRE ATT&CK technique-by-technique playbooks, community Sigma rules, or published least-frequency-analysis patterns ("stack-count parent process for cmd.exe, look at the long tail"). Data collection is broad (endpoints, network, cloud, SaaS audit logs), and the query layer (DuckDB, ClickHouse, the SIEM itself) can answer the questions the playbooks ask.

This is, by the surveys I've seen and by the consensus in my own consulting interviews, where the majority of organizations with a named threat-hunting function actually sit, and SANS surveys put it somewhere north of half, though I think the real figure may be higher once you exclude organizations that claim HMM3 capability based on a single analyst's side project. Either way HMM2 is the modal state for most hunting programs rather than a deficiency, which is worth saying plainly so a team sitting there doesn't read this essay as an indictment.

HMM3 — innovative

The team creates new hunting procedures rather than only applying existing ones. That requires statistics (outlier detection, correlation analysis), linked-data analysis (entity-relationship and graph queries), data visualization beyond the SIEM's defaults, and at least prototype-grade machine learning. The data layer typically grows to include notebook environments (Jupyter, sometimes Databricks), versioning of analytical queries (Nessie or similar), and experiment tracking (MLflow). Hypotheses get framed, tested against historical data, and either validated into a new procedure or discarded. Published outputs include techniques the broader community didn't have before.

HMM4 — leading

Validated HMM3 hunts get automated into production detections. The pipeline is real: feature engineering from the lakehouse, model training and deployment, precision and recall monitoring, drift detection, A/B testing of detection variants. Analyst attention shifts toward novel adversary behavior because the known patterns are handled. The realistic ceiling for this stage isn't 95 percent automation but something closer to 30 to 40 percent of investigation time, and I'll come back to why.

The most common failure mode I see is an HMM2 organization that wants HMM4 automation and tries to buy its way past HMM3. A vendor sells a "ML-driven detection platform" that ships with pre-trained models, and those models generate false positives at a rate that's unworkable for the SOC's analyst capacity, because they were trained on somebody else's environment rather than on this organization's baselines. The SOC has no internal capability to retune them, because that retuning is the HMM3 capability that didn't get developed, so after three to six months the platform gets disabled or relegated to "informational only" status, and the team returns to HMM2 procedures.

This pattern repeats often enough that it deserves a name. I think of it as the "imported HMM4" problem. The infrastructure looks like HMM4 (there's ML in the detection pipeline, there's automation wrapped around it), but the organization's actual capability sits at HMM2, and the gap shows up as unmaintained models drifting toward irrelevance.

The correct sequencing is the boring one. Develop HMM3 capability internally: analysts who can frame and validate hypotheses, who can read a confusion matrix without flinching, who can decide whether a precision drop from 92 to 87 percent is acceptable for a given detection. Then, and only then, invest in the automation pipeline that turns validated hunts into production detections. The order matters because the validation skill is the thing that keeps HMM4 working over time, and buying that skill from a vendor gets you the artifact while leaving the underlying capability undeveloped.

I want to be careful here because the temptation, when writing an essay like this, is to pivot into a tooling sales pitch. The infrastructure section exists because detection engineers need to argue for budget, and the budget argument has to name specific capabilities. But the relationship between infrastructure and maturity runs one direction only. Better infrastructure enables higher maturity if the analyst skills exist; it cannot substitute for those skills.

HMM2 minimum

Broad data collection across endpoints, network, cloud, and SaaS. One to two years of retention on a lakehouse table format (Apache Iceberg is what I default to, for reasons I've written about elsewhere). A query engine that doesn't force the analyst into the SIEM's dialect; DuckDB and ClickHouse both work, depending on workload shape. Threat intelligence integration. The ability to apply community detection rules (Sigma) and MITRE ATT&CK-derived hunting procedures without quarterly license negotiations to add a new log source.

HMM3 additions

Notebook environments wired to the lakehouse. Jupyter against DuckDB is the cheapest starting point I know of, and it scales further than most teams expect. Versioning for analytical queries so experiments are reproducible (Nessie if you've already adopted Iceberg). Experiment tracking (MLflow) for the ML prototypes that HMM3 work generates. And, above all, time, because the dollar cost of the platform is comparatively small while the protected-analyst-time cost is the line item that gets cut first even though it matters most.

HMM4 additions

A real MLOps pipeline. Model deployment infrastructure (Kubernetes plus a serving layer, or a managed equivalent). Continuous monitoring of precision, recall, and detection drift. Feedback loops where analyst dispositions update training data. A/B testing infrastructure for detection variants. The engineering investment is substantial (typically two to three dedicated MLOps engineers, plus the infrastructure they run on), and that's before you count the data-engineering work to keep the feature pipelines stable. Eighteen to twenty-four months from "we have HMM3 capability" to "we have a production HMM4 pipeline" is the timeline I see most often, and it's still aggressive.

The abstract version of the HMM2-to-HMM3 transition is easy to nod along to and hard to plan against. A more concrete version helps. Take a SOC running broad EDR, network, and cloud audit-log collection into an Iceberg lakehouse, with DuckDB available for analyst queries and ClickHouse fronting the production dashboards. The team applies MITRE ATT&CK procedures and community Sigma rules competently. They are unambiguously HMM2.

An HMM3 transition for that team starts with a hypothesis the team generates themselves. Something like: "When an attacker compromises a service account, the credential's usage pattern shifts in measurable ways within the first twenty-four hours: different source IP distribution, different time- of-day distribution, different downstream API call mix." That hypothesis is grounded in security intuition but not derived from a published playbook. The team has to validate it against their own data, and that validation is the HMM3 skill being exercised.

Validation looks like: pull six months of service-account authentication events from the lakehouse into a notebook. Engineer features for each account-day, such as source-IP entropy, hour-of-day distribution Wasserstein distance from the previous week, API call mix Jaccard similarity. Label the small set of known-compromised account-days from past incident reports. Build a simple model (logistic regression is a reasonable starting point; the team should resist the urge to reach for gradient boosting until they understand why the simple model fails). Compute precision and recall against held-out data. Decide, with calibrated reasoning, whether the model is worth promoting to a hunt procedure other analysts can run.

That sequence of steps (frame, engineer, train, evaluate, decide) is the HMM3 daily-practice skill, and it isn't exotic, because data scientists in other industries do it before lunch. The reason it's rare in security operations is that traditional security training paths don't include it, and the organizational reward structures in most SOCs penalize the experimentation overhead it requires, so twenty hours spent on a hypothesis that doesn't pan out reads, on the weekly metrics, as twenty hours of unproductive analyst time. Until that reward structure changes, HMM3 work tends to happen despite the organization rather than because of it.

The teams I've seen make the transition successfully do two things consistently. They set aside explicit experimentation time (typically one day per analyst per week) and they create a lightweight write-up format for failed experiments. The failed-experiment write-up matters because it makes the experimentation work visible to leadership as work, rather than invisible as absence of a deliverable. Once leadership can see the experimentation loop running, the budget conversation changes shape.

Self-assessment against the HMM ladder is more useful when the questions are uncomfortable. The ones I ask in scoping conversations:

• When the team runs a hunt this month, where did the hunting procedure come from? If the honest answer is "MITRE ATT&CK, a vendor blog, or a community Sigma rule," that's HMM2. If the answer is "an analyst on this team developed it and validated it against our environment," that's HMM3.
• Can the team articulate, in writing, the difference between a true positive and a true positive that generalizes? HMM3 work depends on that distinction; HMM2 work doesn't require it.
• What percentage of analyst time, protected and on the calendar, is allocated to experimentation that may not produce a deliverable? If it's zero, the team is structurally HMM2 regardless of what the tooling looks like.
• When a vendor demo shows ML-driven detection, can the team articulate what they'd need to see to trust the model in their environment? HMM2 teams can identify the marketing claim. HMM3 teams can name the validation experiment they'd run.
• If a detection in production starts firing twice as often as it did last month, can the team tell you in an afternoon whether the change is signal or noise? That's the daily-operations test of HMM3 versus HMM2.

Few if any of those questions appear in the vendor maturity assessments I've seen, and they're the ones that distinguish organizations whose maturity matches their stated level from organizations whose maturity matches their tooling.