Schema-on-read vs schema-on-write. The $250K question.

Splunk stores raw events essentially unprocessed in compressed journal files within time-bounded buckets (hot/warm/cold storage tiers). During indexing, Splunk extracts only five default fields: _time, host, source, sourcetype, and index.

TSIDX files associate keywords with event locations, enabling fast keyword-based filtering. But structured field queries (user_id=alice, status_code=404, source_ip=192.168.1.1) must parse raw text on every search using regex-based field extraction rules.

An authentication log event, raw storage:

2024-01-15T14:23:47Z auth_server user=alice@company.com src_ip=203.0.113.45 result=failed reason=invalid_password attempt=3

Field extraction, applied at search time:

| rex field=_raw "user=(?<user_id>[^\s]+)"
| rex field=_raw "src_ip=(?<source_ip>[^\s]+)"
| rex field=_raw "result=(?<auth_result>[^\s]+)"

This regex parsing executes every time you query these fields, not once at ingestion.

The compute cost compounds.

For a single ad-hoc search, parsing overhead is negligible. For dashboards refreshing every 5 minutes and correlation rules running continuously, costs compound dramatically. An authentication failure dashboard counting failed logins per user across 24 hours scans 50 GB of logs daily, runs regex operations across 50 million events, and refreshes 288 times per day. At Splunk Cloud's $0.50–1.50/query pricing, that single dashboard costs about $288/day, or $8,640/month.

Multiply across a typical SOC and the arithmetic runs away: 20 dashboards × $8,640 = $172,800/month, and 100 correlation rules running continuously adds another $200,000+/month, which puts the upper bound around $370,000/month in repeated parsing costs once you count the whole estate rather than one panel.

Splunk's answer: Data Model Acceleration.

Splunk's answer to repeated parsing is Data Model Acceleration (DMA), essentially materialized views for TSIDX summaries. Define a data model specifying field extractions and structure, enable acceleration (Splunk pre-computes TSIDX summaries in background), and query using tstats instead of search.

Performance impact documented by Hurricane Labs: unaccelerated search 10.232 seconds, accelerated tstats query 0.038 seconds, a 270× speedup. Splunk Enterprise Security relies heavily on accelerated data models for correlation searches; without acceleration, ES correlation rules would timeout or consume excessive resources.

But DMA carries its own costs. Storage overhead consumes 10–50% of raw data size, the CPU overhead for background summarization competes with production workloads, acceleration gaps appear when summarization jobs timeout or fail, and the schema rigidity means changes to a data model require full re-acceleration. Splunk's own documentation warns that "index-time custom field extraction can degrade performance at both index time and search time by enlarging indexes," which is to say that pre-extracting fields at ingestion (moving toward schema-on-write) creates its own performance challenges inside Splunk's architecture.

Enterprise security operations ingest logs from 100–200 different sources. Endpoint vendors (CrowdStrike, SentinelOne, Defender, Carbon Black) each invent their own field names. Network vendors (Palo Alto, Cisco, Fortinet, Check Point) have zero standardization. Cloud providers (AWS CloudTrail, Azure Activity, GCP Audit) have three different schema patterns. SaaS vendors (Okta, Office 365, Salesforce, Workday) each invent their own structure.

Field name variations for "source IP address":

Schema-on-write approach: map every vendor's field to a standardized schema at ingestion (OCSF, ECS, CIM). Schema-on-read approach: ingest as-is, handle field variations at query time:

| eval source_ip=coalesce(RemoteIP, SrcIpAddr, src, sourceIPAddress, caller_ip_address, 'client.ipAddress')

Operational flexibility: schema-on-read allows ingesting new log sources immediately and exploring data before committing to schema design. Schema-on-write requires 2–3 days of parser development per source before ingestion.

Retroactive field extraction during investigations.

Threat hunting scenario: investigating suspected data exfiltration, you hypothesize DNS tunneling is the method. You need to extract the query_length field from DNS logs to identify unusually long queries (>100 characters).

Schema-on-read (Splunk):

index=dns
| rex field=_raw "query=(?<dns_query>[^\s]+)"
| eval query_length=len(dns_query)
| where query_length > 100

Result: extract new field retroactively from 90 days of DNS logs, no reindexing required.

Schema-on-write (Elasticsearch): if query_length wasn't in your original index mapping, you have two options. Runtime field (Elasticsearch 7.11+) calculates the field at query time using Painless script (performance penalty similar to schema-on-read). Or reindex 90 days of DNS logs with new field mapping (hours to days of work).

The operational reality is that investigations evolve unpredictably, so the fields you end up needing tend to emerge from patterns you find during analysis rather than from a schema you defined in advance, and that flexibility is much of why a schema-on-read SIEM stays dominant in threat hunting even though it runs 3–10x more expensive than schema-on-write alternatives.

CIM as middle ground.

Splunk's Common Information Model provides standardized field names across data sources while maintaining schema-on-read flexibility. Vendors provide "add-ons" that map their raw formats to CIM fields at search time rather than index time, so searches reference CIM field names (src_ip, dest_ip, user) regardless of vendor while the underlying raw data stays unmodified, which is what lets vendor-neutral detection rules travel across SIEMs.

Limitations: field extraction still happens at search time (repeated parsing overhead remains). Not all vendors provide CIM-compliant add-ons. CIM coverage is incomplete (~40 data models, doesn't cover all log types). Elastic Common Schema (ECS) and OCSF provide similar standardization but require schema-on-write normalization at ingestion, not search-time mapping.

This essay frames schema-on-read vs schema-on-write as a binary architectural choice and resolves it with a hybrid pattern. Iceberg V3 introduces a third option that partially dissolves the binary: the Variant type (semi-structured/nested JSON, ratified in Parquet August 2025) lets you have schema-on-read semantics inside a schema-on-write storage format. Concretely: you can store a CloudTrail event as (eventTime TIMESTAMP, eventName STRING, raw_event VARIANT) in an Iceberg table, query the structured fields with predicate pushdown, AND retain the full nested JSON for retroactive field extraction during investigations. See the Iceberg V3/V4 recalibration essay for the longer take.

Variant doesn't eliminate the "$250K remapping" problem this essay describes. It doesn't eliminate the need to maintain OCSF-normalized projections for cross-vendor query consistency. But it does substantially change the cost calculation. The raw-retention layer no longer requires a separate Splunk-style schema-on-read system, and the normalized projection no longer has to be exhaustive; you can OCSF-project the top-50 fields and leave the long tail in the Variant column for hunting-tier access.

Adoption caveat: V3 Variant engine support is still rolling out across Spark, Trino, DuckDB, and Snowflake through 2026. The hybrid architecture this essay recommends is still the right answer if your engine doesn't support variant_get() yet.

The second read/write axis: schema is upstream of commit.

Everything above is about one read/write question, the schema one: do you parse at ingest or parse at query. Once you put the data in a lakehouse, a second read/write question shows up underneath it, and the two are easy to conflate. I borrow the cleaner version of this distinction from Roman Kolesnev at Streambased, whose post "Your Iceberg table doesn't need to exist" splits the table interface into two contracts. The first is a read contract. Hand an engine a table identifier and it gets back a schema and a set of scannable bytes, and it does not care whether a Spark job wrote those bytes, a database inlined them, or they were synthesized on demand from a Kafka topic. Every realization preserves that read contract; preserving it is the point of the interface, and it is why the raw-Variant column above works as the hot/virtual realization point behind a single read contract. The second is a write contract, or commit contract: how data actually enters the table.

The commit contract is where the realizations diverge, and where streaming-ingest economics ride. Classic Iceberg writes Parquet files and registers them through the catalog, which is durable and engine-agnostic but slow, and it write-amplifies badly at high commit rates because each commit drops its own metadata footprint. DuckLake commits a SQL transaction instead, gets ACID from the database's existing concurrency control, and inlines small batches rather than spawning a file per write. Streambased's ISK never writes at all, because producing to the Kafka topic is the write. Committing every few seconds is exactly where Iceberg's per-commit file footprint turns into a $/GB problem, so for a high-frequency detection-ingest path the commit contract, not the schema question, is the axis that sets the bill. The schema read/write choice this essay spends most of its length on sits upstream of, and separate from, the commit-contract choice. You can decide parse-at-ingest versus parse-at-query without yet deciding file-write versus SQL-transaction versus never-write, and the cost surprises tend to hide in the second decision.

The honest null on all of this: a V4-efficient materialized Iceberg backend, single-file commits and Parquet-for-metadata landing cleanly, might be good-enough from the hot detection tier through the multi-year audit tier, in which case a tiered or polyglot substrate is complexity I talked myself into rather than complexity the workload demanded. I lean toward tiering being real for security data, where sub-second streaming detection and seven-year retention pull in genuinely different directions, but that is a lean, not a proof, and the null deserves a fair test rather than a rhetorical dismissal.

So I ran the part of that test I could run cleanly on a single host. On a MOAR reference stack against real MinIO object storage (the bench script is lab/commit_tax.py), I wrote the same 100,000 rows two ways, once as a single batch commit and once as a 100-commit stream, which is the shape a streaming detection-ingest path actually has. Classic Iceberg held the prediction almost exactly: the metadata footprint went from 8.9 KB to 4,579 KB across the streamed version (about 515×), planning time rose from 8.7 ms to 181 ms (about 21×), ingest stretched from 0.44 s to 16.3 s (about 37×), and the data file count went from one to a hundred, because every commit drops its own file and its own metadata whether the batch is large or tiny. DuckLake ran the identical 100-commit stream as SQL transactions, kept its metadata in the catalog database, held planning flat at roughly 7 ms, and inlined the small commits so they produced zero Parquet files. The absolute milliseconds are a single-host artifact and I wouldn't quote them as a benchmark, but the shape is the finding, and the shape is Iceberg's file-per-commit floor against DuckLake's catalog-transaction commit, which is the commit contract moving the streaming bill rather than the schema-on-read-versus-write choice that sits upstream of it.

Choose schema-on-read when:

• 100+ diverse data sources with inconsistent field naming
• Exploratory threat hunting is a core operational requirement
• Retroactive field extraction during investigations is common
• New log sources appear frequently (M&A, vendor changes)
• Small-to-medium scale (<500 GB/day where costs are manageable)
• Splunk expertise already exists and transition costs are prohibitive

Cost tolerance: accept 3–10× higher costs for operational flexibility.

Choose schema-on-write when:

• Known detection patterns dominate (dashboards, compliance, scheduled queries)
• Schema stability exists (mature security stack, infrequent log source changes)
• Cost optimization is the priority (budget-constrained, high volumes)
• Cloud-native environment (Azure/GCP with free ingestion for native logs)
• Data engineering expertise available (to build normalization pipelines)

Cost benefit: 60–75% savings versus schema-on-read at scale.

Choose hybrid (raw + normalized) when:

• Petabyte-scale volumes make pure schema-on-read cost-prohibitive
• Both hunting and detection are critical operational requirements
• OCSF/ECS/ASIM adoption provides a stable normalization target
• Data engineering team can build and maintain dual storage
• 1–2 year timeline acceptable for migration with parallel operation

Sweet spot: 1 TB/day to 10 TB/day organizations with mature security operations.

Schema-on-read buys operational flexibility and pays for it in query performance and license spend. That is the right bargain for threat hunting, where the investigation path is unpredictable by definition and the field you need is the one nobody indexed. Schema-on-write inverts the bargain. It is faster and cheaper per query but rigid against schema change, which is exactly what you want for known detection patterns and compliance reporting and exactly what you don't want the morning a new log source lands mid-incident.

The honest answer for most teams over 1 TB/day with the data-engineering staff to run it is to stop choosing. Keep both: raw retention on cheap object storage for the hunting tier, OCSF-normalized projection on warm for detection. The framing has two layers, though, and that is the part I would send a reader away holding. The schema question, parse-at-ingest versus parse-at-query, is the one this essay measures. Underneath it sits the commit-contract question, file-write versus SQL-transaction versus never-write, and on a high-frequency streaming-ingest path that second layer is where the $/GB bill is actually set. Decide them separately. Most cost surprises I have seen came from teams that solved the schema question carefully and then let the commit contract default to per-commit file writes at a few-second cadence.

Chronicle's BigQuery backend, Sentinel's ASIM normalization, OCSF's vendor-neutral schema, and Iceberg V3's Variant type are all converging on the same shape: keep the raw bytes, project the fields you query, and don't pretend one paradigm covers every workload. I'd build for both layers and resist the tidy single-substrate story until V4's commit-footprint work has been tested against a real seven-year-retention, sub-second-detection security estate rather than a slide. The first slice of that test is now on the board: the commit-tax bench above measures classic Iceberg paying a roughly 515× metadata and 37× ingest penalty across a 100-commit stream while a catalog-transaction commit stays flat, so the commit-contract prediction is no longer a lean, it is observed on a real object store. What is still untested is whether a V4-efficient materialized Iceberg backend closes that gap well enough to carry the whole range from hot detection to multi-year audit on one substrate, and until that larger test runs, tiering is the bet I'm making, and I'm telling you it's a bet.