Kafka to Iceberg: the integration hidden costs.

Apache Kafka is optimized for streaming: append-only writes, offset-ordered reads, small messages on the order of 16 KB, short retention measured in days to weeks, throughput delivered through partition parallelism. It is, by design, not a data warehouse.

Apache Iceberg is optimized for analytics. Large scans with column pruning and predicate pushdown, Parquet files in the hundreds-of-megabytes range, retention measured in months to years to decades, partitioning by business dimensions (event time, asset, severity) rather than offset order.

The integration question is what happens at the seam, because Kafka's natural output unit, a 16 KB message on a partition, is roughly ten thousand times smaller than Iceberg's natural input unit, a 512 MB Parquet file, and bridging that gap is an architectural decision that determines whether your lakehouse stays performant or drowns in small files.

The vendor pitches around this seam are loud ("zero-copy! Single source of truth! No duplication!"), and they obscure four practical costs that, in my experience and in the practitioner discussions I trust, are what determine whether the integration succeeds.

Once you accept that compaction has to run, the next question is whether it can keep up. Compaction throughput (measured in megabytes per second of small files rewritten into large ones) has to exceed the ingest write rate, on average, across the time window you care about. If it doesn't, the small-file backlog grows monotonically and query performance degrades on a schedule.

The Kafka Connect Iceberg sink (the open-source connector originally from Tabular, now widely deployed) lands data as small files on the streaming side and relies on a separate Spark or Flink job to compact them. That separation is operationally clean: the streaming side stays fast, the compaction side runs on a budget you control. The pitfall is that the compaction job is sized independently, and a quiet schema change or a sudden traffic spike can push it under capacity without any obvious alert until query latency degrades.

Confluent's TableFlow (the managed Kafka-to-Iceberg path inside Confluent Cloud) and Aiven's Iceberg Topics (the equivalent on Aiven's managed Kafka) bundle the compaction job into the managed service. That's convenient. It's also where the pricing gets opaque. Compaction compute is part of the bill; how much you pay depends on workload shape rather than a flat per-GB rate. I have not seen a public methodology disclosure from either vendor that lets me model compaction cost ahead of time. Treat their pricing as something to measure in a paid pilot, not something to model from the rate card. (Pricing claims here are directional; I have not independently verified 2026 quotes.)

For my own H-REALTIME-OCSF-01 proof-of-concept (Kafka Connect Iceberg sink writing OCSF Network Activity events to Iceberg through a Nessie REST catalog, on a single-node WSL2 environment), I measured roughly 7,000 events per second sustained, with zero data loss, before compaction lag became the binding constraint. That's a constrained environment and production hardware would scale higher, but it establishes a copy-based baseline I trust because I ran it myself, and what ran out first was compaction throughput rather than Kafka write rate or Iceberg metadata throughput.

Kafka treats schema as an immutable contract. The schema registry says "this is what producers write and consumers expect," and the value of that contract is that it does not change underneath downstream consumers. Iceberg treats schema as evolvable: add a column, drop a column, rename a column, all without rewriting historical files.

Those two stances are not incompatible, but they require coordination. When the EDR vendor adds a new field to the event format, somebody has to decide whether the field flows into the Iceberg table as a new column, whether older Iceberg rows backfill with null, and whether downstream dashboards and detection rules need to be updated before or after the schema change lands. With copy-based integration (Kafka Connect Iceberg sink, Confluent TableFlow, Aiven Iceberg Topics), the materialization layer is the natural place to handle this, so you transform the schema during the write, add the column to Iceberg without touching Kafka, and keep the two evolutions independent.

With zero-copy integration (shared storage between Kafka and Iceberg), the coordination problem gets harder. Zero-copy systems including Bufstream, Aiven's tighter-shared-storage modes, and StreamNative's Ursa share Parquet files between the streaming layer and the analytics layer. That shared storage means you cannot evolve the Iceberg schema without affecting Kafka's view of the data, which leaves two bad options. Option A is the uber-schema approach where every change adds new nullable columns and the Parquet files bloat with deprecated fields. Option B is migrate-forward, where you rewrite historical files with the new schema and break Kafka's "what you wrote is what you read" guarantee, which matters for audit and forensics.

This is the schema-evolution argument Jack Vanlightly (distributed-systems specialist, formerly RabbitMQ, now Confluent) raised in his October 2025 critique of zero-copy Kafka-Iceberg integration, and it is the one critique I think still holds in full. Vanlightly's broader claim, that zero-copy shifts costs from storage to compute by an order of magnitude, has been weakened by the Ursa benchmarks below, but the schema-evolution coordination tax is structural rather than a benchmark question, so copy-based integration sidesteps it.

The version of this essay I would have written a year ago framed zero-copy as vendor marketing wrapped around an architectural compromise, and that framing was reasonable in 2024, but it has weakened since.

Ursa is a lakehouse-native streaming engine published at VLDB 2025 by Merli et al., affiliated with StreamNative and the Apache Pulsar community. The architecture is zero-disk, leaderless, and Kafka-compatible, writing directly to Iceberg or Delta Lake on object storage with no local disk tier. The published numbers (Tier A, peer-reviewed venue): 5 GB/s sustained throughput for both publish and consume, invariant across 6-to-48 partitions and 1 KB-to-64 KB message sizes; P99 publish latency under one second at a 1:5 write-to-read fan-out ratio; reported 92% cost reduction versus disk-based Kafka and 78% versus Kafka's own tiered-storage option; infrastructure spend at roughly 5% of a comparable Kafka stack; CPU utilization at 30 to 60% under peak load with network I/O as the bottleneck rather than CPU.

The CPU number is the one that carries the most weight here, because Vanlightly's October 2025 critique relied on a claim that generating Parquet from Kafka log segments costs "at least an order of magnitude more CPU cycles" than copying log segments to object storage. I could not find a peer-reviewed source for that ten-times figure when I went looking, and the Ursa benchmarks show a network-bound architecture rather than a CPU-bound one. The directional intuition behind the critique may still be correct for some workloads; the specific ten-times number should be treated as practitioner estimate, not measured fact.

What Ursa does not validate: it inherits Kafka's offset-ordered partitioning, so the partition inflexibility critique still applies. If your security analytics need partitioning by (event_time, endpoint_id, event_type) for threat hunting, Ursa does not give you that shape. The VLDB paper does not address the schema-evolution conflicts above. And the benchmarks come from StreamNative-affiliated authors; independent multi-org production validation would strengthen the claims considerably.

The honest update is narrow: peer-reviewed benchmarks now exist for a zero-copy architecture writing to open table formats, and they show economics that may justify accepting the partition and schema trade-offs for greenfield deployments. That is a different claim than "zero-copy is ready for security analytics." For brownfield Kafka deployments with custom partitioning needs, copy-based remains the safer call.

Apache Iceberg V3 introduces row-level lineage (row IDs plus last-updated tracking), which enables CDC (change data capture) directly from Iceberg without external metadata. For Kafka-to-Iceberg integrations carrying update streams (rather than append-only telemetry), V3 row lineage may materially reduce the compute cost of keeping a streaming-fed Iceberg table consistent over time, because point updates become metadata operations rather than full snapshot diffs.

For most security telemetry (EDR events, firewall logs, CloudTrail, network flow records) the workload is append-only and the V3 row lineage advantage is small. Where it may matter: asset inventory tables, user-identity tables, threat-intel feeds that update in place. Those are legitimately CDC-shaped and V3's mechanics may simplify the integration meaningfully. I cover the mechanics in more depth in Row lineage and detection engineering →.

The hedge is that engine support for V3 features is still rolling out across Spark, Trino, DuckDB, and Snowflake through 2026, and production references for streaming-fed Iceberg tables using V3 row lineage are sparse as of early 2026, so the capability exists in the spec while the operational maturity is still forming, which means you should verify your engine version before assuming V3 mechanics in production. None of this solves the fundamentals above (small files, compaction, exactly-once, schema evolution), because those are the structural costs of bridging streaming and analytics, and they persist regardless of V3.

Choose copy-based if

• You already operate Kafka and migration costs are the binding constraint.
• Your security analytics require custom partitioning by event time, asset, or severity.
• Schema evolution is frequent: quarterly EDR updates, OCSF revisions, new cloud audit fields.
• Compliance and forensics workflows need immutable Kafka history plus an evolvable Iceberg view.
• Operational clarity matters, with separate teams owning streaming and analytics layers.

Tooling: Kafka Connect Iceberg Sink for the open-source path. Confluent's TableFlow if already on Confluent Cloud. Aiven's Iceberg Topics in copy-based mode if already on Aiven. Compaction job (Spark or Flink) sized to exceed average write rate by a comfortable margin.

Evaluate zero-copy seriously if

• Greenfield deployment with no existing Kafka commitment; partition strategy is still open.
• Storage cost is the primary constraint and the Ursa-class economics are credible for your scale.
• Offset-ordered partitioning is acceptable for your analytics workload.
• Schema is stable, with well-defined data contracts and infrequent revisions.

Tooling: Ursa (StreamNative, open formats, VLDB-published benchmarks). Bufstream (Buf Technologies, dual-mode zero-copy plus copy-based as of late 2024). Aiven's tighter Iceberg Topics modes. Tabular File Loader is the open-source path for getting batched files into Iceberg without a Kafka intermediary at all, which is sometimes the right answer for non-streaming pipelines that have been forced through Kafka unnecessarily.

Kafka-to-Iceberg integration is more than a wiring exercise, because the four hidden costs (small files, compaction throughput, exactly-once delivery, schema-evolution coordination) are themselves the architecture. The copy-based versus zero-copy debate is a debate about which trade-offs you're willing to absorb on which workloads, and the honest answer in 2026 is workload-by-workload rather than platform-wide.

The Ursa VLDB paper is the strongest update to this picture in two years, and while it does not make zero-copy the right call for every security workload, it does make "zero-copy is just vendor marketing" an outdated framing. Iceberg V3 row lineage adds another option for the small subset of workloads that are CDC-shaped, but the fundamentals are unchanged, because bridging streaming and analytics still requires compaction, exactly-once, and schema coordination.

For the security teams I work with, copy-based Kafka Connect Iceberg Sink remains the default recommendation, with compaction compute as the line item I push hardest to size correctly, because the duplication cost is small while the compaction cost is the one that determines whether the lakehouse stays performant, so it is the number worth sizing carefully.