Format deep-dive

Variant type ends the flattening wars.

Apache Iceberg V3 adopts a semi-structured column type called Variant, ratified in the Parquet specification in August 2025. For security teams running into the kind of detection-rule failure I documented in the flattening anti-pattern (the CloudTrail MFA-bypass bug where a SOC team's detection rule silently returned the wrong answer for six weeks) Variant is the architectural fix that V2 didn't have. It removes the choice between correctness and query-engine portability, which were the two horns of that dilemma.

Reading time: about 15 minutes. Evidence tier: B for the V3 Variant specification mechanics (Apache Parquet Variant type ratified August 2025, Apache Iceberg V3 specification). C for the working CloudTrail example below; it's representative of the schema, not pulled from a production deployment. The detection rule failure mode is from a real, anonymized post-mortem. Engine support for V3 Variant is rolling out across 2026 and I hedge specific claims accordingly.

The recap

The six-week detection outage.

A SOC team migrated from Splunk to a ClickHouse-based lakehouse. They had a detection rule that fired on IAM privilege-escalation events without multi-factor authentication, which is exactly the kind of rule you want firing if an attacker is inside your AWS environment. The Splunk version of the rule worked. The flattened ClickHouse version of the rule returned zero results for six weeks. An internal review later found 23 actual events during that window that should have alerted.

The root cause was schema translation, not engine choice. CloudTrail's mfaAuthenticated field is only present when MFA was used. Splunk's schema-on-read architecture (where the raw JSON is preserved and parsed at query time) kept this distinction intact. A query checking WHERE userIdentity.sessionContext.attributes.mfaAuthenticated="false" returned the events where MFA was attempted and failed. A query checking WHERE NOT(mfaAuthenticated="true") returned the events where MFA was not used at all, because the field was absent from the original payload, so these were two different security questions that needed two different queries and returned two different answer sets.

The flattened ClickHouse schema (one column per JSON path, materialized at ingest) treated the absence of the field as NULL. The naive translation of the detection rule into WHERE userIdentity_sessionContext_attributes_mfaAuthenticated = 'false' returned only the rare case where MFA was attempted and failed. It missed all 23 cases of "no MFA at all." The detection logic hadn't broken in any way the engineer could see; it had been translated into the wrong question.

The fix at the time was ClickHouse Nested types, a ClickHouse-specific feature that preserves hierarchical structure and lets queries express "field absent" semantics correctly. This worked. It also locked the team into ClickHouse for that table: the same Iceberg table couldn't be queried from Trino, DuckDB, or Snowflake without re-implementing the nested-extraction logic in each engine. The team had traded a correctness failure for a portability failure, and V3 Variant is what lets you avoid both at once.

The mechanism

What the Variant type actually does.

Variant is a column type that stores semi-structured data (typically JSON, though the format is more general) natively inside a Parquet file. "Semi-structured" in this context means data with a schema that can vary between rows: each event may have a different set of fields, fields may be nested at varying depths, and the presence or absence of a field is itself meaningful. CloudTrail, Azure Activity logs, Okta system logs, Kubernetes audit logs, and most cloud-native telemetry are semi-structured by nature. The four properties that make Variant useful here:

Field-presence semantics are preserved by construction. A Variant column stores the actual JSON object, including the distinction between "field absent," "field present with value NULL," and "field present with an explicit value." A query against the Variant column can ask any of these three questions and get predictable answers. This is the property the flattened ClickHouse schema lost when it collapsed every JSON path into a top-level column.

I wanted to make that claim falsifiable rather than argue it from reasoning alone, so I built a small DuckDB demonstrator (it runs as lab/variant_mfa.py against the MOAR reference stack on a single host) that isolates the absence-versus-NULL mechanism on a synthetic corpus of 820 ConsoleLogin events: 500 with mfaAuthenticated=true, 120 with mfaAuthenticated=false, and 200 where the field is absent entirely (a legacy or mis-instrumented path that never asserted MFA at all). The security-correct count of logins not protected by MFA is the false events plus the absent ones, so 320. A flattened detection written naively as mfa = 'false' returns 120 and silently misses the 200 absent-MFA logins, which are the riskiest of the set because nothing ever attested to a second factor; a structure-aware query that tests the nested path catches all 320. The demonstrator confirms what the flattened schema can't express: it collapses "absent" into the same NULL as "present-but-null," so the naive flattened rule under-counts unprotected logins by 200. This is a synthetic demonstrator on a single host, built to isolate the mechanism rather than to report a production incident count, but it does show the failure is reproducible and not just a story about one post-mortem.

Predicate pushdown still works. Queries against Variant fields use a variant_get(column, '$.path.to.field', 'TYPE') syntax (JSON path extraction with a typed return) that the query engine can push down to the file format level. Iceberg's metadata pruning, meaning the min/max statistics it keeps in manifest files, can be computed for variant paths so that files which can't possibly match the predicate are skipped without being read. This is the property that storing JSON as an opaque STRING column always lost.

The same column may be queryable from multiple engines. This is the headline claim, and it's also where I want to hedge most carefully. The Parquet Variant specification was ratified in October 2025, and the Iceberg V3 specification adopted Variant, but engine support is rolling out unevenly, on no single schedule, and not always with feature parity, so I'll get specific about the adoption state in a moment.

You don't have to choose between flattening and storing as opaque BLOB. The two V2-era options for nested data were "flatten and lose semantics" or "store as JSON STRING and lose predicate pushdown." Variant gives you both, because the structure is preserved, the column stays queryable, and the file format keeps enough information to skip irrelevant files, which is the part that changes the architectural conversation for security data.

The fix, in SQL

The CloudTrail MFA rule, translated to Variant.

The fix is short enough to fit in two code blocks. Here is the broken V2 / flattened version that missed 23 events:

CREATE TABLE cloudtrail_v2 (
    eventTime TIMESTAMP,
    eventName STRING,
    userIdentity_userName STRING,
    userIdentity_sessionContext_attributes_mfaAuthenticated STRING,  -- often NULL
    -- ... 50+ other flattened fields
    requestParameters_userName STRING,
    requestParameters_policyArn STRING
) USING iceberg;

-- Detection rule: returns ZERO results for "no MFA at all" events
SELECT eventTime, userIdentity_userName, eventName
FROM cloudtrail_v2
WHERE eventName IN ('AttachUserPolicy', 'PutUserPolicy', 'AddUserToGroup')
  AND userIdentity_sessionContext_attributes_mfaAuthenticated = 'false';
-- Catches "MFA attempted, failed"
-- Misses 23 actual privilege escalations without MFA

And here is the V3 / Variant version that captures both failure modes correctly:

CREATE TABLE cloudtrail_v3 (
    eventTime TIMESTAMP,
    eventName STRING,
    userIdentity VARIANT,         -- preserves the entire nested object
    requestParameters VARIANT,
    sourceIPAddress STRING,
    awsRegion STRING
) USING iceberg
TBLPROPERTIES ('format-version' = '3');

-- Detection rule: correctly captures both "no MFA" failure modes
SELECT
    eventTime,
    variant_get(userIdentity, '$.userName', 'STRING') AS userName,
    eventName,
    variant_get(requestParameters, '$.policyArn', 'STRING') AS policyArn
FROM cloudtrail_v3
WHERE eventName IN ('AttachUserPolicy', 'PutUserPolicy', 'AddUserToGroup')
  AND variant_get(userIdentity, '$.sessionContext.attributes.mfaAuthenticated', 'STRING') IS NULL;
-- IS NULL correctly captures "field absent = no MFA used"
-- Catches all 23 events that the V2 query missed

The total schema change is to replace the 50+ flattened columns with two Variant columns, and the total query change is to replace the underscore-delimited column names with variant_get() calls, while the detection logic (the predicate that says "this is an IAM privilege-escalation event without MFA") is otherwise unchanged. The absence-versus-NULL distinction that broke the V2 rule is recovered for free, because Variant preserved the original JSON object rather than projecting each path into its own column.

Engine support

Who supports Variant in early 2026.

This is the section I want to be most careful about. The Parquet Variant specification was ratified in October 2025. The Iceberg V3 specification adopted Variant. Engine support is rolling out unevenly across 2026, and the gap between "the spec supports it" and "your production engine version supports it" is exactly where migration pain hides. The state as of early 2026, based on project documentation and vendor announcements I've been able to verify:

Apache Spark 4.1: the most complete support I've seen. variant_get() and Variant-aware predicate pushdown both work in the Spark SQL surface that Iceberg V3 tables expose.
DuckDB: partial support. Reading Variant columns works in recent releases; write support is rolling out. If your pipeline produces data with Spark and queries it with DuckDB, you may be fine today; the reverse direction is less certain.
Trino: on the roadmap, expected in the second half of 2026 based on the project's public discussions. I would not bet a production migration on Trino Variant support landing on a specific date.
Snowflake: has had a native VARIANT type since before the Parquet specification existed. The native type maps to the Iceberg V3 Variant type on read; write support against Iceberg V3 Variant from outside Snowflake requires a connector path that's still maturing.
ClickHouse: community Iceberg integration exists. Variant support specifically is earlier-stage; I have not been able to confirm a production-ready release path as of this writing, and I'd treat it as "verify against your specific version" rather than "assume it works."

The honest version of the multi-engine claim is that the Parquet and Iceberg specifications make it possible for multiple engines to read the same Variant column with consistent absence-versus-NULL semantics, while the engine implementations are catching up to the specification on different schedules. If you're planning a migration today that depends on cross-engine Variant queries, the answer to "which engines work?" is "the specific versions of Spark and DuckDB you're running, verified by hand, plus whatever your connector path to Snowflake looks like," which is better than V2, where the answer was "pick one engine and lock yourself in," though it isn't yet "every engine, everywhere, today."

The variant_get() syntax itself also varies slightly between engines. Spark's Variant functions, DuckDB's JSON-path syntax, and Snowflake's : path notation are similar in intent but not bit-identical in spelling. If you're writing a detection-rule library that's meant to run against multiple engines, expect to maintain a thin translation layer for the path expressions, because the semantics (what the query asks) are portable even though the exact tokens you type may not be.

What it doesn't fix

Four things Variant is not.

The post-DETECTION-13 conversation surfaced misreadings of what Variant can do. Four things worth being explicit about, because each one is a place where I've seen the capability oversold.

1. Variant doesn't replace OCSF normalization.

The point of OCSF (the Open Cybersecurity Schema Framework) is cross-vendor query consistency. A detection rule written against class_uid = 3001 (the OCSF class for Account Change events) should fire whether the underlying log is CloudTrail, Azure Activity, an Okta system log, or a GitHub audit event. Variant gives you a cross-engine way to store the original vendor-native event without losing semantics. OCSF gives you a cross-vendor way to query normalized events without per-vendor field-name knowledge. Both are useful and they compose: store the raw event in a Variant column, project the OCSF fields into structured columns, and let the analyst pick which to query depending on whether they need raw fidelity or normalized portability.

2. Variant doesn't make detection latency faster.

This piece is about correctness, not latency. The CloudTrail detection rule above runs in the hunting tier (queries that scan minutes of recent history, not millisecond-budget streaming alerts). For detection-tier alerting, the same logic needs to live in the streaming layer (Flink, RisingWave, pipeline-based detection running before the data lands in Iceberg). Variant doesn't change that, and it doesn't promise a streaming-tier latency improvement.

3. Variant doesn't fix all flattening problems.

Some security data is naturally tabular (Zeek conn.log records, Suricata flow records, EDR process events) and flattening it doesn't lose anything because there's nothing nested to lose. Variant is for the case where the source format is nested and field presence carries information. For naturally tabular sources, flattening is still the right answer. Wrapping a flat record in a Variant column would add overhead without recovering any semantic that was already in the column types.

4. Variant doesn't eliminate schema drift.

If CloudTrail adds a new field to the userIdentity object next year, your existing Variant queries don't break. The new field is just present in the column, queryable via variant_get(). That's a real win compared to flattened schemas, which require an ALTER TABLE every time a vendor adds a field. But your detection rules don't automatically know about the new field. If the new field has security relevance (say AWS adds an MFA-method field that distinguishes hardware tokens from SMS), someone has to write the detection logic that uses it. Variant makes the storage layer schema-flexible. The detection-engineering process is still a human in the loop.

Migration path

How to move a flattened CloudTrail schema to Variant.

If you're currently running a flattened CloudTrail schema and the failure mode above is something you recognize (or worse, something you've already shipped to production and haven't caught yet), here's the phased migration I'd recommend. The phases are designed so each one stands on its own; you can stop at the end of any phase if priorities shift.

Phase 1: add a Variant column alongside (one to two weeks)

Don't drop the flattened columns yet. Add a single Variant column to the existing table that captures the original nested userIdentity and requestParameters objects. Most ingestion pipelines that produce flattened CloudTrail are doing so by parsing the original JSON; change them to also write the original JSON to the new Variant column. You now have both representations in the same row, which lets you compare the new and old query paths against the same underlying events.

ALTER TABLE cloudtrail_v2 ADD COLUMN userIdentity_variant VARIANT;
-- Backfill via Spark: read original JSON from S3 archive,
-- populate userIdentity_variant for the retention window.

Phase 2: rewrite detection rules in parallel (two to four weeks)

For each detection rule that depends on field-presence semantics (the CloudTrail MFA case is the canonical example, but every nested log type has a few), rewrite the rule to use variant_get() against the Variant column. Run the new rule and the old rule side by side for at least one cycle and compare results. Expect the new rule to find events the old rule missed. Investigate each one to confirm the new rule is correct rather than over-firing.

This phase is where most of the value shows up, because what you're recovering is the set of detections that were silently broken rather than the schema change itself, so treat the parallel run as a detection-engineering exercise and not merely a schema migration.

Phase 3: drop the flattened columns, optionally (four to eight weeks)

Once all consuming queries have migrated to the Variant column, you can drop the flattened columns to reclaim storage and simplify the schema. This is optional. There's no harm in keeping both representations side by side, and the storage cost of the flattened columns is small relative to the Variant column. The reason to drop them is documentation hygiene: a schema with one VARIANT column beats a schema with fifty flattened columns plus one VARIANT column for reasoning about what the table actually contains.

Engine-version caveat before you start

V3 Variant requires variant_get() and Variant-aware predicate pushdown in your query engine. Before phase 1, verify the specific version of every engine that reads this table. If your primary query engine is Trino and Trino Variant support hasn't landed in your version yet, hold off on the migration until it ships, because you don't want to migrate to a Variant schema and then discover that the engine your analysts actually use can't query it efficiently, and the cost of waiting is small next to the cost of half-migrating.

The bigger frame

Why the flattening anti-pattern existed in the first place.

Flattening wasn't a mistake people made out of carelessness. It was the rational response to a constraint that V2 imposed: if you wanted predicate pushdown, columnar compression, and engine portability, you had to project nested JSON paths into top-level columns at ingest. The alternative (store the JSON as an opaque STRING column) gave up predicate pushdown and made every query a full-table scan with runtime JSON parsing. Neither option was good. Flattening was the less-bad option for most workloads, and the CloudTrail MFA case was the price you paid when field presence carried information.

This connects to a broader pattern I've written about in schema-on-read versus schema-on-write. Schema-on-write systems (flattened ClickHouse, traditional data warehouses, anything that projects fields into typed columns at ingest) give you fast queries at the cost of locking in the field set you knew about at ingest time. Schema-on-read systems (Splunk, Elastic, raw JSON over a query engine) give you flexibility at the cost of paying parse time on every query. Variant doesn't fully dissolve that binary, but it does partially dissolve it for the specific case where the source format is nested and field presence is semantically loaded. You get schema-on-write performance for the structured columns you've projected out (timestamp, event name, source IP) and schema-on-read flexibility for the nested object underneath.

For the same reason, Variant sits alongside the deletion-vector and row-lineage work that I cover in the Iceberg V3 thesis shift piece, where the three together read less like a single feature and more like a recalibration of which trade-offs you have to make when you choose Iceberg. Variant is the piece that matters most for security data specifically, because security telemetry is overwhelmingly semi-structured.

The coda

The six-week detection outage, replayed under V3.

The SOC team in the original story switched to ClickHouse Nested types, which fixed their immediate correctness problem but locked them into ClickHouse for that table. Two years later (in this hypothetical replay) they want to move some of their CloudTrail analysis to DuckDB for cost reasons, and they discover that the Nested-typed schema doesn't translate cleanly. They face a second migration. The original incident cost them six weeks of missed detections. The portability lock-in will cost them another quarter of engineering time when the cost-optimization pressure forces the engine change.

If V3 Variant had existed when they did the original migration, the second migration may not have been necessary. The same Iceberg table would be queryable from ClickHouse, DuckDB, Trino, and Spark, with consistent absence-versus-NULL semantics across all of them, subject to the engine-version caveats above (which is why I say "may" rather than "would"). The CloudTrail MFA detection rule would run anywhere the team chose to point its analysts.

So the case for Variant comes down to this, that it ends the choice between correctness and engine portability, which were the two horns of the dilemma the flattening anti-pattern documented, and you no longer have to pick between them. The catch is that "ends the choice" is an architectural claim about the specification, while "ships in your production engine today" is an operational claim about your specific Spark, DuckDB, Trino, or Snowflake version, so verify the second before betting on the first.