Technology deep-dive

Sigma 2.0 correlations and the pySigma backend reality.

I wrote a foundational piece on Sigma as the fourth open standard in the security lakehouse and deliberately left the correlation question hedged. This essay does the unhedged version. I name the four Sigma 2.0 correlation types, walk through the pySigma backend ecosystem with its community-published maturity states, and describe what conversion quality looks like when you compile real rules to real engines. The correlation extension is improving but lags vendor-native correlation engines materially, and the backend conversion story is "useful per backend, with caveats" rather than "write once, deploy everywhere." I also compiled a rule set to four backends, reported below, to put a first-party measurement under the per-backend claims rather than leaning only on the community maturity signal.

Reading time: about 19 minutes. Evidence tier: B overall (Sigma specification documents, the pySigma plugin directory's published state field, vendor product pages I can verify) with Tier A on the correlation specification itself, and a first-party compilation benchmark (reproducible, pinned versions) added below. Flagged inline.

The correlation specification defines four types, each with a structured YAML shape, a required timespan, optional group-by fields, and (for the count types) a condition comparison operator. The four types are:

1. event_count

Tallies how many times a referenced atomic Sigma rule fires inside a defined time window, grouped by whatever fields you specify. This is the "five failed logins in sixty seconds" idiom. The YAML references one or more underlying Sigma rules by ID, sets a timespan (the spec accepts compact strings like 60s, 5m, 1h, 1d), defines a group-by field set (typically a user, source IP, or host identifier), and applies a comparison operator from gt, gte, lt, lte, eq, neq against a threshold count.

title: Failed-login burst from same source
id: 9d0e8e0a-...-correlation
correlation:
  type: event_count
  rules:
    - failed_login_event
  group-by:
    - SourceIP
  timespan: 60s
  condition:
    gte: 5

2. value_count

Counts distinct values in a specified field within the timespan, grouped by the group-by set. The canonical use is enumeration detection: one source IP attempting authentication against many distinct usernames in a short window, or one host resolving an unusually large number of distinct domains. The shape is the same as event_count with an additional field attribute naming the field whose distinct values are being counted.

3. temporal

Fires when all referenced rules match within the same timespan and group-by scope, regardless of order. This expresses "any combination of these N indicators inside this window, attributable to the same entity," useful for kill-chain-style detections where ordering is unknown or noisy. No threshold condition is required; the assertion is set-membership across the rules list.

4. temporal_ordered

Same as temporal with the additional constraint that the referenced rule matches must occur in the listed order. This is the kill-chain progression idiom: initial access, followed by discovery, followed by lateral movement, attributable to the same entity, inside a defined window. It's the most expressive of the four types and also, predictably, the one with the most variance in backend support quality.

Two structural notes that matter for portability. Correlation rules reference atomic rules by ID, which means your atomic rule library is the base layer and your correlation rules are the analytic layer above it. Correlations can also be chained (a correlation rule can reference another correlation rule's ID) which gives composability for higher-order detections. That second property is powerful in principle and sparsely supported in practice; chained correlations are where backend coverage starts to fragment.

The spec covers the four idioms that compose most multi-event SOC detections. What it does not cover is statistical baselining ("this account's login rate is three standard deviations above its trailing thirty-day baseline"), graph-shaped entity correlation, or join-heavy multi-source enrichment beyond the field-equality group-by mechanism. Those remain vendor-native territory, and an architect committing to Sigma needs to know where the abstraction ends.

The backend-support claims above come from the plugin directory's state field, which is the community's maturity signal rather than a measurement, so I compiled a small rule set and looked at what each backend actually emits. Eleven Sigma rules — six single-event detections and five correlation rules covering all four types — compiled to four open backends (Splunk SPL, Elasticsearch ES|QL, Elasticsearch Lucene, and OpenSearch PPL) on pySigma 1.3.3, with the compiler run twice and asserted byte-identical so the result reproduces. The harness and the verbatim queries are public.

The single-event rules ported everywhere, six of six on all four backends, which is the clean-conversion result the next section describes, confirmed rather than asserted. The correlation rules are where the four backends separated, and they separated in three different ways.

Elasticsearch Lucene refused all five, raising NotImplementedError: Backend does not support correlation rules. A filter-only query language cannot express aggregation, so the backend raises rather than emit a query that means less than the rule intended, and that is the safe way to fail, because you find out at compile time that the rule did not translate. Splunk SPL and ES|QL sat at the other end: both preserved the count, distinct-count, and unordered-temporal correlations in full, with the aggregation, the threshold, and the time window all present in the generated query, and both refused temporal_ordered, the ordered-sequence type, which neither implements at these versions. So the read above on ES|QL as one of the stronger correlation targets holds up under measurement, with the ceiling at ordered sequencing.

OpenSearch PPL is the one that should make you careful, because it translated every correlation type, including the ordered sequence the other two refuse, and that breadth hides a loss: on the count-based correlations, the brute-force and spray detections, it drops the time window. The same brute-force rule compiles two ways:

# Splunk SPL — the five-minute window is in the query
... | bin _time span=5m
    | stats count as event_count by _time TargetUserName
    | search event_count >= 10

# OpenSearch PPL — same rule, no window
... | stats count() as event_count by TargetUserName
    | where event_count >= 10

The SPL query buckets into five-minute windows and counts within each; the PPL query counts across the whole search range, so a threshold of ten failed logons meant to fire on ten-in-five-minutes instead fires on ten-ever. The query is syntactically valid, it runs, and it looks like a working brute-force detection, and nothing errors, which is what makes the loss easy to miss. PPL keeps the window on the temporal rules, where it emits span(@timestamp, 5m), so the drop is specific to its count path rather than a blanket limitation, which is exactly the per-construct unevenness a single directory state field cannot show you. Across the five correlation rules that is three brute-force and spray detections compiling to queries that lost their time-bounding with no warning.

The boundary on this measurement matters: it is what the compiler emits, not what the SIEM executes, so a window absent from the query might be supplied by a dashboard time range or a scheduled-search interval, and a present construct is not proof of correct execution. But the emitted query is the artifact a practitioner copies into a SOC, so a dropped window is one someone has to remember to add back. The checks are disclosed and every verbatim query is recorded in the methodology, and a newer backend release can move any of these cells, so the honest use of the result is to pin the versions and re-run it, which the harness makes a single command.

The pySigma project publishes a plugin directory (pySigma-plugin-directory on the SigmaHQ GitHub organization) that lists every recognized backend with a community-assigned state. The states are defined in the directory's README as stable (working state, maintained), testing (working but unfinished), devel (under development), broken (dysfunctional but maintained), and orphaned (unmaintained). Treating that field as the closest thing to a community maturity signal (Tier B, with the obvious caveat that "stable" doesn't mean "feature-complete for correlations") here is the matrix I'd use to plan a pilot.

Stable backends I'd pilot first

• Splunk (SPL and tstats). The flagship stable backend. Compiles atomic rules to both plain SPL and tstats data-model queries, with savedsearches.conf output for deployment. The most production-mileage of any pySigma target. The correlation story here is the closest to parity with the vendor-native equivalent, because SPL has the most expressive query language to compile into.
• Microsoft Kusto (KQL). Marked stable. Covers Microsoft Sentinel, Microsoft 365 Defender (XDR Advanced Hunting), and Azure Data Explorer. The Sentinel ASIM mapping is part of the toolchain. A credible target for atomic-detection portability. Correlation feature support in the KQL backend continues to improve and varies by release; verify against the current package version.
• Elasticsearch (Lucene, ES|QL, EQL). Stable. The ES|QL output is the most strategically interesting because Elastic's piped-query language is closer in shape to Sigma's correlation model than legacy Lucene. The backend has been updated to include ES|QL correlation support, which makes it one of the stronger correlation-extension targets I'm tracking.
• Loki (LogQL). Maintained by Grafana, marked stable for atomic rules. The correlation support has been a tracked open issue on the project's GitHub. Treat the atomic-rule coverage as production-credible and the correlation coverage as not-yet-production for this backend specifically.
• IBM QRadar (AQL). Stable. Compiles to AQL queries. Useful for organizations with QRadar-anchored SOCs that want to author detections in a portable layer.
• Palo Alto Cortex XDR (XQL). Stable. Compiles to XQL queries against Cortex XDR.
• Carbon Black (queries for EDR), SentinelOne (Deep Visibility, PowerQuery), Rapid7 InsightIDR (LEQL). All marked stable in the plugin directory. These EDR-focused backends matter because they extend the Sigma authoring surface into endpoint detection content where the vendor-native query languages are less standardized than SPL or KQL: a useful place for an open IR.
• OpenSearch (Lucene). Stable, with a maintenance status that tracks the Elasticsearch backend. Important for organizations on AWS OpenSearch Service.
• Panther. A pySigma backend named for the Panther detection-as-code platform exists in the plugin directory marked stable, which is a useful signal that the vendor takes the Sigma IR seriously enough to support a compilation target.
• Logpoint. Stable. Logpoint has publicly described their pySigma backend in product marketing.

Testing-state backends: useful but verify

• PowerShell, HAWK.io, Datadog Cloud SIEM, SQLite (and Zircolite), ClickHouse (clicksiem's pysigma-backend-clickhouse, added since my May check), NetWitness, SurrealQL, Golang Expr. All listed in the testing state. "Working but unfinished" per the directory's own definition. For pilot environments these are interesting; for production-critical detection paths I'd verify the specific feature coverage before committing, especially for the correlation extension.

Development-state backends: exploratory only

• Google SecOps (formerly Chronicle): UDM search and YARA-L 2.0. This is the one that surprises people. Chronicle is one of the most commonly cited Sigma targets in vendor materials, but the official pySigma SecOps backend is listed in the devel state in the plugin directory as of mid-2026. That doesn't mean it's unusable; it means the community maturity signal is "under development, not stable." If you're standardizing on Chronicle and Sigma, verify the current state of the backend against your specific rule set before committing to a pure pySigma path. Vendor-supplied or partner-supplied conversion paths may be more mature than the open pySigma backend.
• Trellix Helix, Quickwit, STIX, Azure Log Analytics (ala-socprime), OSSEM pipeline. Development-state. Treat as experimental.

Conspicuously absent

StarRocks, DuckDB, and a dedicated Trino or Presto backend remain absent: no pySigma backends in the SigmaHQ plugin directory for any of them as of my recheck in June 2026, so the canonical compilation path an Iceberg-anchored program would want still isn't there. Two narrow things did move since my May check, though. ClickHouse now has a community backend in the directory, clicksiem's pysigma-backend-clickhouse (on PyPI at 1.0.0), carrying the testing state; it compiles atomic rules to ClickHouse SQL, but I haven't found documented Sigma-correlation support, so for a correlations essay it sits in the same atomic-yes, correlation-unproven place as most testing-state backends. And there is a single-author pySigma-backend-athena under the SigmaHQ org, not listed in the plugin directory, that emits Presto/Trino-compatible SQL and, alone among the backends I've read, transpiles the event_count correlation into a real SQL window function. The author notes it likely works against any Trino engine, but it is a one-contributor, 1-star repo, only event_count is implemented (value_count, temporal, and temporal_ordered raise NotImplementedError), and the Trino compatibility is untested, so I read it as a proof-of-concept worth watching rather than a backend to standardize on. For lakehouse-anchored detection the practical workarounds are still (a) compile to the dictquery or sqlite backends and adapt the SQL output, (b) use the Elasticsearch ES|QL output as a piped-query starting point, or (c) maintain native SQL for multi-event correlation and use Sigma for the atomic layer. None of these are clean.

The thinness of mature pySigma backends for the lakehouse engines I'd otherwise recommend is still the most consequential gap in the Sigma portability story for an Iceberg-anchored security architecture, though it is now a narrowing gap rather than an empty one: a testing-state ClickHouse backend and an event_count-only Athena/Presto proof-of-concept are early, partial answers that stop well short of the maintained, full-correlation backends the architecture actually wants. Worth tracking as a 2026–2027 development frontier, and a real reason to pair atomic detections in Sigma with engine-native authoring on the correlation tier for now.

The SigmaHQ public rule repository is the largest open detection ruleset in security, with thousands of community-contributed YAML rules organized by log source and tagged against MITRE ATT&CK techniques. It also drives the loudest criticism of Sigma. Community contribution at scale produces a long tail of low-quality rules that an organization can easily import wholesale and regret. The honest version: many community rules are noisy, many encode environment-specific assumptions that don't generalize, and many are stale. That is true. It is also true for every open detection ruleset that has ever existed.

Nasreddine Bencherchali, one of the SigmaHQ maintainers, has written publicly about the project's quality assurance pipeline. Every pull request runs through validators, then a "good-log test" against the SigmaHQ evtx-baseline repository that flags rules generating false positives against known-good telemetry, then a regression test that requires submitters to contribute a sample malicious log validating the rule's usefulness. Rules that fail any gate do not merge. That is genuinely more rigor than I've seen in any other open detection community.

The "bad rules, good defaults" framing I land on: the repo has rules of widely varying quality, and the repo as a whole encodes good detection defaults that an organization can lean on rather than reinvent. The discipline you need on top of the repo is curation: pick the rules that match your environment, your log sources, your tolerance for false positives, not blind import.

The other underappreciated property: ATT&CK technique tagging is consistent and machine-readable. An architect can mechanically compute coverage maps from compiled rule sets, identify under-covered techniques, and drive detection-engineering priority from coverage gaps. That mechanical-mapping property is worth more than the marginal quality of any individual community rule, and it's the foundation of the work I describe in detection maturity.

It is worth being precise about how cross-SIEM coverage actually gets solved in practice today, because the marketing framing implies a translation engine that doesn't exist, and the working pattern is aggregation rather than auto-translation. Look at Michael Haag's Security Detections MCP, which aggregates more than 8,200 community detections, where the method is to gather content per SIEM and index the whole thing by MITRE ATT&CK technique rather than to take one canonical rule and auto-translate it across every backend. Coverage comes from curating a rule per platform and tying them together through the ATT&CK tag, with the technique ID as the join key rather than a compiler, and that distinction matters for an architect because per-backend curation plus ATT&CK indexing is the pattern that works at scale right now. Atomic auto-translation through pySigma works for the bread-and-butter shapes, but correlation auto-translation is the part that breaks down, which is exactly why the corpus is organized around aggregation rather than a single portable source rule.

I keep landing on a hybrid recommendation in this essay because the honest answer is that the choice is per-rule, not per-program. Here is the heuristic I use.

Author in Sigma when

• The rule is atomic, a single-event match against known indicators or fingerprints.
• The rule is going to be deployed to more than one engine (a primary SIEM plus a long-tail archive, a cloud SIEM plus an on-prem SIEM, an EDR query language plus a SIEM correlation).
• The rule is a candidate for community sharing or community sourcing, because your organization wants to contribute it upstream, or you're sourcing it from SigmaHQ in the first place.
• The correlation pattern is event_count or value_count and the target backend has stable correlation support for the type you need.
• The detection lifecycle expects engine churn, whether a SIEM migration in the next eighteen months, a planned lakehouse adoption, or a dual-vendor procurement cycle.

Keep vendor-native when

• The detection relies on statistical baselining, anomaly scoring, or graph-shaped correlation that the Sigma correlation extension does not express.
• The detection uses vendor-specific enrichment, lookup macros, or threat-intel join patterns that the IR doesn't have a clean idiom for.
• The detection is a high-value temporal_ordered correlation with three or more sequenced rules and the target backend's coverage of that idiom is testing-state or worse.
• The detection lives entirely inside one engine and the engine isn't moving in the foreseeable planning horizon.
• The performance characteristics of the compiled Sigma output don't meet the SOC's response-time requirements, and the vendor-native equivalent runs materially faster.

The choice is rarely "go all-in on Sigma" or "stay on vendor-native," because the two will coexist in most production SOCs for the foreseeable future, and the boundary between them moves over time as the correlation extension matures and as backend coverage firms up.

Anvilogic and Panther (the two commercial products I named in the foundational piece) both lean into this hybrid framing in their public product documentation. Anvilogic exposes a Sigma-compatible authoring surface alongside vendor-native authoring paths and compiles to multiple SIEM backends. Panther maintains its own pySigma backend in the SigmaHQ plugin directory (stable state) alongside Python-based rule authoring. Neither claims pure Sigma implementation. Tier B evidence: I've read the vendor materials and verified the Panther backend in the plugin directory; I have not run either product in production.

Four moves, ordered from cheapest to highest-investment, that test whether Sigma 2.0 plus pySigma fits your environment without committing to a wholesale migration.

• Audit your detection portfolio by shape. Categorize every active detection as atomic, event_count, value_count, temporal, temporal_ordered, statistical-baselining, or graph-correlation. The first five are Sigma's surface; the last two are not. Bucket sizes tell you the theoretical Sigma coverage before you compile a single rule. If 70% of your detections are atomic and event_count, Sigma is a high-leverage adoption. If 70% are statistical-baselining, it is not.
• Pilot pySigma on ten atomic rules across two backends. Pick two engines you already maintain duplicate detections in (commonly Splunk plus one other). Pick ten stable, important atomic rules. Author them in Sigma. Compile to both. Compare false-positive and false-negative parity against your hand-maintained versions over a two-week window. The signal is conversion quality on your rules, not on demo rules.
• Test one event_count correlation per backend. Author a single event_count correlation you can validate against ground truth, such as a known failed-login burst or a known enumeration attempt. Compile, run, and compare against the vendor-native equivalent on the same telemetry window, since that is the smallest test that tells you whether the correlation extension is production-credible for your backend version.
• Stand up the CI/CD pipeline before scaling rule count. The biggest mistake I see is authoring a hundred Sigma rules before having lint, regression test, and per-backend compilation in CI. The pipeline is the work that makes Sigma pay off; the rules are the easy part. Build against the ten pilot rules first, then scale. Pair with the DetectFlow pattern.

That sequence respects the load-bearing hedge: Sigma 2.0 is improving, the backend ecosystem is uneven and worth verifying per release, and the correlation extension is lagged by vendor-native engines on the hardest detections. None of these are reasons to skip Sigma; they are reasons to size the adoption work honestly before the rule count compounds into a maintenance burden.