Schema-on-read vs schema-on-write. The $2.7M question.

Splunk stores raw events essentially unprocessed in compressed journal files within time-bounded buckets (hot/warm/cold storage tiers). During indexing, Splunk extracts only five default fields: _time, host, source, sourcetype, and index.

TSIDX files associate keywords with event locations, enabling fast keyword-based filtering. But structured field queries (user_id=alice, status_code=404, source_ip=192.168.1.1) must parse raw text on every search using regex-based field extraction rules.

An authentication log event, raw storage:

2024-01-15T14:23:47Z auth_server user=alice@company.com src_ip=203.0.113.45 result=failed reason=invalid_password attempt=3

Field extraction, applied at search time:

| rex field=_raw "user=(?<user_id>[^\s]+)"
| rex field=_raw "src_ip=(?<source_ip>[^\s]+)"
| rex field=_raw "result=(?<auth_result>[^\s]+)"

This regex parsing executes every time you query these fields, not once at ingestion.

The compute cost compounds.

For a single ad-hoc search, parsing overhead is negligible. For dashboards refreshing every 5 minutes and correlation rules running continuously, costs compound dramatically. An authentication failure dashboard counting failed logins per user across 24 hours scans 50 GB of logs daily, runs regex operations across 50 million events, and refreshes 288 times per day. At Splunk Cloud's $0.50–1.50/query pricing, that single dashboard costs about $288/day, or $8,640/month.

Multiply across a typical SOC: 20 dashboards × $8,640 = $172,800/month. 100 correlation rules running continuously adds another $200,000+/month. Total: $370,000+/month in repeated parsing costs at the upper bound. This is the repeated parsing problem at enterprise scale.

Splunk's answer: Data Model Acceleration.

Splunk's answer to repeated parsing is Data Model Acceleration (DMA) — essentially materialized views for TSIDX summaries. Define a data model specifying field extractions and structure, enable acceleration (Splunk pre-computes TSIDX summaries in background), and query using tstats instead of search.

Performance impact documented by Hurricane Labs: unaccelerated search 10.232 seconds, accelerated tstats query 0.038 seconds — a 270× speedup. Splunk Enterprise Security relies heavily on accelerated data models for correlation searches; without acceleration, ES correlation rules would timeout or consume excessive resources.

But DMA has significant limitations. Storage overhead consumes 10–50% of raw data size. CPU overhead for background summarization competes with production workloads. Acceleration gaps appear when summarization jobs timeout or fail. Schema rigidity means changes to a data model require full re-acceleration. Splunk's own documentation warns that "index-time custom field extraction can degrade performance at both index time and search time by enlarging indexes." Translation: pre-extracting fields at ingestion (moving toward schema-on-write) creates its own performance challenges in Splunk's architecture.

Enterprise security operations ingest logs from 100–200 different sources. Endpoint vendors (CrowdStrike, SentinelOne, Defender, Carbon Black) each invent their own field names. Network vendors (Palo Alto, Cisco, Fortinet, Check Point) have zero standardization. Cloud providers (AWS CloudTrail, Azure Activity, GCP Audit) have three different schema patterns. SaaS vendors (Okta, Office 365, Salesforce, Workday) each invent their own structure.

Field name variations for "source IP address":

Schema-on-write approach: map every vendor's field to a standardized schema at ingestion (OCSF, ECS, CIM). Schema-on-read approach: ingest as-is, handle field variations at query time:

| eval source_ip=coalesce(RemoteIP, SrcIpAddr, src, sourceIPAddress, caller_ip_address, 'client.ipAddress')

Operational flexibility: schema-on-read allows ingesting new log sources immediately and exploring data before committing to schema design. Schema-on-write requires 2–3 days of parser development per source before ingestion.

Retroactive field extraction during investigations.

Threat hunting scenario: investigating suspected data exfiltration, you hypothesize DNS tunneling is the method. You need to extract the query_length field from DNS logs to identify unusually long queries (>100 characters).

Schema-on-read (Splunk):

index=dns
| rex field=_raw "query=(?<dns_query>[^\s]+)"
| eval query_length=len(dns_query)
| where query_length > 100

Result: extract new field retroactively from 90 days of DNS logs, no reindexing required.

Schema-on-write (Elasticsearch): if query_length wasn't in your original index mapping, you have two options. Runtime field (Elasticsearch 7.11+) calculates the field at query time using Painless script (performance penalty similar to schema-on-read). Or reindex 90 days of DNS logs with new field mapping — hours to days of work.

Operational reality: investigations evolve unpredictably. The fields needed emerge from data patterns discovered during analysis, not from predefined schemas. This flexibility is why Splunk remains dominant in threat hunting despite being 3–10× more expensive than schema-on-write alternatives.

CIM as middle ground.

Splunk's Common Information Model provides standardized field names across data sources while maintaining schema-on-read flexibility. Vendors provide "add-ons" that map their raw formats to CIM fields at search time, not index time. Searches reference CIM field names (src_ip, dest_ip, user) regardless of vendor. Underlying raw data remains unmodified. Vendor-neutral detection rules become portable across SIEMs.

Limitations: field extraction still happens at search time (repeated parsing overhead remains). Not all vendors provide CIM-compliant add-ons. CIM coverage is incomplete (~40 data models, doesn't cover all log types). Elastic Common Schema (ECS) and OCSF provide similar standardization but require schema-on-write normalization at ingestion, not search-time mapping.

This essay frames schema-on-read vs schema-on-write as a binary architectural choice and resolves it with a hybrid pattern. Iceberg V3 introduces a third option that partially dissolves the binary: the Variant type (semi-structured/nested JSON, ratified in Parquet October 2025) lets you have schema-on-read semantics inside a schema-on-write storage format. Concretely: you can store a CloudTrail event as (eventTime TIMESTAMP, eventName STRING, raw_event VARIANT) in an Iceberg table, query the structured fields with predicate pushdown, AND retain the full nested JSON for retroactive field extraction during investigations. See the Iceberg V3/V4 recalibration essay for the longer take.

Variant doesn't eliminate the "$2.7M remapping" problem this essay describes — it doesn't eliminate the need to maintain OCSF-normalized projections for cross-vendor query consistency. But it does substantially change the cost calculation. The raw-retention layer no longer requires a separate Splunk-style schema-on-read system, and the normalized projection no longer has to be exhaustive — you can OCSF-project the top-50 fields and leave the long tail in the Variant column for hunting-tier access.

Adoption caveat: V3 Variant engine support is still rolling out across Spark, Trino, DuckDB, and Snowflake through 2026. The hybrid architecture this essay recommends is still the right answer if your engine doesn't support variant_get() yet.

Choose schema-on-read when:

• 100+ diverse data sources with inconsistent field naming
• Exploratory threat hunting is a core operational requirement
• Retroactive field extraction during investigations is common
• New log sources appear frequently (M&A, vendor changes)
• Small-to-medium scale (<500 GB/day where costs are manageable)
• Splunk expertise already exists and transition costs are prohibitive

Cost tolerance: accept 3–10× higher costs for operational flexibility.

Choose schema-on-write when:

• Known detection patterns dominate (dashboards, compliance, scheduled queries)
• Schema stability exists (mature security stack, infrequent log source changes)
• Cost optimization is the priority (budget-constrained, high volumes)
• Cloud-native environment (Azure/GCP with free ingestion for native logs)
• Data engineering expertise available (to build normalization pipelines)

Cost benefit: 60–75% savings versus schema-on-read at scale.

Choose hybrid (raw + normalized) when:

• Petabyte-scale volumes make pure schema-on-read cost-prohibitive
• Both hunting and detection are critical operational requirements
• OCSF/ECS/ASIM adoption provides a stable normalization target
• Data engineering team can build and maintain dual storage
• 1–2 year timeline acceptable for migration with parallel operation

Sweet spot: 1 TB/day to 10 TB/day organizations with mature security operations.

Schema-on-read optimizes for operational flexibility at the cost of query performance and expense. For threat hunting workloads where investigation paths are unpredictable, this trade-off is justified.

Schema-on-write optimizes for query performance and cost efficiency at the cost of schema rigidity. For known detection patterns and compliance reporting, this trade-off is optimal.

Hybrid architectures provide the best of both worlds for organizations with sufficient data engineering expertise and scale (1+ TB/day) to justify the operational complexity.

The $2.7M question isn't "which approach is better?" but rather "which problems am I solving, and what costs am I willing to accept?" For petabyte-scale security data infrastructure prioritizing manageability over extreme performance, dual storage (raw + normalized) is the emerging best practice; OCSF/ECS/ASIM normalization is a strategic hedge against vendor lock-in; selective materialization (dashboards, compliance) beats universal acceleration; 6-month parallel operation during migration is what makes the change tractable; and schema-on-read preservation matters for threat hunting even in schema-on-write architectures.

The platforms advancing state-of-the-art — Chronicle's BigQuery lakehouse, Sentinel's ASIM normalization, OCSF's vendor-neutral schema, Iceberg V3's Variant type — all point toward hybrid approaches rather than religious adherence to one paradigm. Build for both. Optimize where it matters. Preserve flexibility where investigation demands it.