Iceberg vs Delta Lake for security data.

This essay was originally drafted assuming Iceberg V2 mechanics, where the "Iceberg merge-on-read is less mature than Delta" critique and the "compliance-first means choose Delta for GDPR erasure" recommendation were both defensible. Apache Iceberg V3 changes this materially through three key features:

• Puffin-based deletion vectors replace V2's position-delete and equality-delete files. This is the same mechanism Delta has used for its merge-on-read maturity advantage. The "Iceberg MoR is immature" claim was a V2 statement.
• Row-level lineage (row IDs plus last-updated tracking) enables incremental processing and CDC-from-Iceberg without external metadata, weakening Delta's Change Data Feed advantage.
• Variant type (semi-structured / nested JSON, ratified August 2025 in Parquet, adopted in Iceberg V3) reframes the schema-evolution conversation for security log ingestion.

Three caveats apply. Engine support for V3 features is rolling out across Spark, Trino, DuckDB, and Snowflake through 2026, so verify your engine version before assuming V3 mechanics in production. Delta Lake has a longer track record on these capabilities, which still matters for risk-averse compliance use cases. The vendor-neutrality and multi-engine arguments in this essay are unaffected by V3, because they turn on ecosystem dynamics rather than table-format mechanics, and they remain the strongest reasons to prefer Iceberg.

Specific in-line updates appear below the V2-era claims they affect. The original analysis is preserved so you can see what changed.

This essay assumes dedicated security data infrastructure, which is a separation-of-duties best practice. When security data lives on isolated infrastructure, separate from corporate data platforms containing PII and financial data, many architectural decisions simplify.

Why isolation matters for table-format choice

Network isolation plus IAM as the security boundary. Dedicated security VPC/VNet, team-only access (no cross-functional multi-tenancy), simplified security posture with the network boundary plus IAM providing the primary control.

Encryption overhead becomes optional. Shared platforms must encrypt Iceberg metadata (10–20% query overhead) when PII or financial data mixes with security logs. Isolated platforms can treat metadata encryption as optional, performance-first.

RBAC complexity reduces. Shared platforms need fine-grained row-level security and column masking. Isolated platforms can use table-level permissions, avoiding the 5–30% RLS latency tax.

Compliance requirements simplify. Operational security logs (EDR telemetry, network flows, cloud audit trails) have lower compliance burden than PII. Isolation satisfies separation requirements without HIPAA/PCI-DSS-grade hardening on the security plane.

Production validation: Netflix runs dedicated observability infrastructure (5 PB/day, isolated from production systems). Huntress runs a dedicated security platform (3M endpoints, 93% cost reduction, isolated). Jake Thomas at Okta runs isolated analytics infrastructure (7.5T records, dedicated to the security team). The full sources for the Huntress and Okta numbers are on their teardown pages.

When isolation assumptions don't hold (shared corporate data platform with security data mixed with finance and operations, multi-tenant security teams like an MSSP managing 50+ customer tenants, or PII / financial data in security logs which is rare but possible for fraud-detection workloads), compliance shifts and you enable all the hardening measures, which favors Unity Catalog plus Delta Lake for built-in governance.

Adobe Experience Platform: 5,000+ tables (2024)

Adobe Experience Platform (customer data platform for marketing) runs 5,000+ active Delta tables, terabytes of data ingested daily, petabytes of data managed for customers (Unified Profile offering). Z-ORDER optimization reduced processing time from hours to minutes, ACID transactions enabled concurrent writes from 100+ pipelines, and schema evolution ran without downtime for customer-facing applications, all on a stack of Databricks plus Unity Catalog plus Delta Lake. So Adobe validates Delta Lake at a scale comparable to Netflix's Iceberg, with petabyte-scale data, thousands of tables, and high-concurrency writes.

InMobi: GDPR/CCPA compliance (2023–2024)

InMobi (mobile advertising platform, security-adjacent data privacy) faced GDPR/CCPA right-to-erasure requirements that meant deleting user data across its data lake. The solution was the Databricks Lakehouse Platform with Delta Lake, where Z-ORDER indexing optimized the point deletes, time travel allowed recovery for the 30-day compliance window before permanent deletion, encryption at rest and in transit ran via FIPS 140-validated modules, and audit trails covered data access. InMobi's pattern (Delta plus Unity Catalog plus audit trails) validates compliance-heavy workflows for data-privacy regulations.

Adobe Real-Time CDP graph workload: 250B messages/day

Adobe's Real-Time CDP team migrated 2 petabytes of actively queried data from NoSQL to Delta Lake, with 5,000+ Delta tables for the multi-tenant architecture, close to 250 billion messages/day across regions, and close to 3 trillion changes/day on the Delta tables, which validates Delta Lake for extreme-scale write concurrency. Security data at scale (petabyte-scale ingestion, high write concurrency, multi-tenant isolation) mirrors this workload closely.

1. Query engine ecosystem

Iceberg: universal compatibility across Spark, Trino, Dremio, Flink, Athena, Snowflake, BigQuery, DuckDB, and Presto. Vendor-neutral. Multi-cloud (query AWS Iceberg tables from Azure Databricks via Polaris).

Delta Lake: Spark-optimized, deepest integration with Apache Spark. Expanding ecosystem (Trino, Athena, BigQuery added 2023–2024). Databricks-first, with best performance within Databricks via Unity Catalog optimization.

Decision: choose Iceberg for multi-cloud portability. Choose Delta if Databricks-committed for the deepest Unity Catalog integration.

2. Metadata architecture

Iceberg: distributed metadata in Parquet/Avro manifest files. Query engines read only needed manifests (efficient at millions of partitions). Catalog-agnostic, works with Glue, Unity Catalog, Polaris, and Hive Metastore.

Delta Lake: transaction log in _delta_log/ (JSON + Parquet checkpoints). Spark caching optimizes log reads. Checkpoint every 10 commits.

Benchmark (DataBeans, 2022, Delta 1.0 vs Iceberg 0.13.0): combined load-plus-query time was 1.68 hours for Delta versus 5.99 hours for Iceberg, ~3.5× faster overall (the load-only gap was narrower, ~1.3×). Caveat: the benchmark is Spark-centric and does not reflect Trino, Dremio, or Athena workloads where Iceberg's distributed metadata may outperform Delta's transaction log.

Decision: Delta if Spark-dominant (ETL pipelines, batch). Iceberg if mixed engines (Athena for compliance, Trino for ad-hoc, Spark for ETL).

3. Schema and partition evolution

Iceberg: partition evolution lets you change strategy without rewriting data. Start with daily partitioning, switch to hourly as volume grows. Hidden partitioning means analysts query by timestamp and Iceberg applies partition filters automatically. Add, drop, rename, reorder columns without table rewrites.

Delta Lake: partition changes require rewriting the table. Analysts must specify partition columns in WHERE clauses. Add columns without rewrites, but rename and reorder require table recreation.

-- Iceberg: add hourly partition transform (does not rewrite existing data)
ALTER TABLE security_events.firewall_logs
ADD PARTITION FIELD hours(event_time);

-- Remove daily partition (deprecate, does not delete data)
ALTER TABLE security_events.firewall_logs
DROP PARTITION FIELD days(event_time);

-- Queries automatically use optimal partitioning (hidden)
SELECT * FROM security_events.firewall_logs
WHERE event_time > now() - interval '24 hours';
-- Iceberg uses hourly for new data, daily for historical

Decision: Iceberg if data volume is unpredictable. Delta if partition strategy is stable (daily partitioning sufficient for 5+ years).

4. CDC and streaming integration

Iceberg: Flink integration for streaming writes. Kafka via copy-based approaches (Kafka → Iceberg via Flink/Spark). V2 merge-on-read used position-delete and equality-delete files, operationally heavy at high update rates. V3 introduces Puffin-based deletion vectors that materially close the MoR maturity gap with Delta. V3 row-level lineage (row IDs + last-updated tracking) enables incremental processing and CDC-from-Iceberg without external metadata, weakening the historical "Delta has Change Data Feed, Iceberg does not" advantage.

Delta Lake: Change Data Feed for native CDC support. Iceberg V3 row lineage is the equivalent capability, though Delta's CDF has more production mileage. Mature MERGE INTO for upserts and SCD Type 2. Spark Structured Streaming optimized for Delta.

Decision: Delta for CDC-heavy workloads with a need for production track record. Iceberg for write-once workloads (immutable security logs, audit trails).

5. Multi-format catalogs (Unity Catalog)

Unity Catalog (Databricks, open-sourced June 2024) supports Delta Lake, Iceberg, and Hudi in a single catalog. Managed Iceberg tables via Unity Catalog's Iceberg REST Catalog API (Public Preview, Databricks Runtime 16.4+ LTS). Foreign catalog access lets you query Iceberg tables from AWS Glue, Hive Metastores, and Snowflake via Unity Catalog, and Delta Sharing for Iceberg is in Private Preview.

Significance: Unity Catalog eliminates the "Iceberg OR Delta" binary. Security teams can use Delta for CDC-heavy tables (user inventory, asset tracking) and Iceberg for immutable logs (CloudTrail, firewall, EDR) with a single governance layer. Adopting Databricks reduces table-format lock-in.

1. Databricks-first architecture

If you're committed to Databricks for 5+ years, with Unity Catalog governance and MLflow for threat-detection models, then Delta provides the deepest Unity Catalog integration (row-level security, column masking native), Spark performance optimizations (1.7–3.5× faster than Iceberg for Spark workloads), and Delta Sharing for secure external sharing, where the trade-off is accepting ecosystem lock-in in exchange for the strongest performance within that ecosystem.

2. CDC-heavy workloads (with caveat)

These are workloads like user behavior analytics, asset inventory tracking, and CMDB synchronization, where Delta has had Puffin-based deletion vectors longer, MERGE INTO has accumulated more production mileage, and Change Data Feed is production-validated. Iceberg V3 brings deletion vectors and row-level lineage onto parity in the spec, but adoption lags the spec, because engine support for V3 deletion vectors is rolling out across Spark, Trino, DuckDB, and Snowflake through 2026.

The "Iceberg MoR is immature" argument was a V2 statement, and for V3 deployments the gap is materially smaller, so verify your query engine version before assuming V3 mechanics in production.

3. Compliance-first security operations (also with caveat)

Think GDPR right-to-erasure, CCPA deletion, and PCI-DSS masking, where Delta's Z-ORDER indexing optimizes point deletes, Puffin-based deletion vectors make erasure cheap, and time travel plus vacuum handles the compliance retention windows, while Databricks holds FedRAMP, HITRUST, HIPAA, and SOC 2 Type II certifications.

Iceberg V3 closes most of this gap. V3 deletion vectors use the same Puffin-based mechanism, making right-to-erasure a metadata update rather than a full file rewrite. The "compliance-first means choose Delta" guidance was a V2 recommendation, and for V3 deployments where engine support has caught up it's no longer a reason to prefer Delta, because the capability is now present in both. InMobi validated Delta at advertising-platform scale, while equivalent Iceberg V3 production references are still emerging (early 2026), so Delta's longer track record still matters for a risk-averse compliance posture.

In late March 2026, Databricks launched Lakewatch: an open, agentic SIEM built on Delta Lake and Unity Catalog, with AI agents powered by Anthropic Claude. Partners include Cribl, Palo Alto Networks, Okta, Wiz, and Zscaler. This changes the competitive landscape for the Iceberg vs Delta decision in one specific way, which is that choosing Delta Lake now comes with a security-specific product ecosystem. If your organization is Databricks-committed and evaluating SIEM alternatives, Lakewatch means your lakehouse table format and your security detection platform share the same foundation of Delta tables, Unity Catalog governance, and Spark compute, and that shared foundation is a genuine integration advantage.

The flip side is that this deepens the Databricks lock-in case against Delta, because when your table format, governance catalog, compute engine, and security detection platform all come from one vendor, the exit costs compound. Databricks markets Lakewatch as "zero vendor lock-in via open formats," but Delta Lake is Databricks-controlled, and an "open agentic SIEM" built on a single vendor's stack is a different kind of open than Apache Iceberg queried by 15+ independent engines. This is Tier D evidence (vendor launch, no production validation), so treat it as a factor in your framework rather than a resolved answer.

Apache Iceberg and Delta Lake are both production-validated at petabyte scale for security-adjacent workloads, so the choice comes down to strategic priorities rather than technical limits.

Iceberg strengths: universal query engine compatibility (15+ engines), vendor neutrality, partition evolution, hidden partitioning for analyst accessibility.

Delta Lake strengths: Spark performance optimization (1.7–3.5× faster), CDC maturity, Databricks ecosystem depth, compliance certifications (FedRAMP, HITRUST, HIPAA).

My recommendation: start with Apache Iceberg for vendor neutrality and multi-engine flexibility. Evaluate Delta Lake if Databricks commitment emerges or CDC-heavy workloads dominate.

Unity Catalog's multi-format support (Delta + Iceberg + Hudi) means this isn't a permanent decision, so choose based on your organization's strategic priorities rather than on technical benchmarks alone.