Iceberg table maintenance at scale.

Apache Iceberg V3 shipped Iceberg-side in 2025 (releases v1.8.0 through v1.10.0), so Snowflake's catalog-side V3 adoption in May 2026 was not the V3 release but a major engine catching up to a spec already in production at other engines. V4 is in active development, and tracking issue #58 on the Iceberg repo is the place to watch for milestone movement. Engine support for V3 features continues to roll out across Spark, Trino, DuckDB, and Snowflake through 2026, so verify your engine version before assuming V3 mechanics in production.

The compaction, snapshot expiration, orphan cleanup, partition evolution, Z-ordering, and lifecycle tiering patterns in this essay are V2/V3 invariant. They target write-amplification problems (small files from streaming inserts, snapshot growth, debris from failed writes) that are independent of the V2-versus-V3 distinction. Two operational areas do change with V3, and both are simplifications rather than new burdens:

• GDPR right-to-erasure workflow. V2 erasure meant writing position-delete or equality-delete files and then compacting them away on a maintenance schedule, which left extra files to manage and extra compaction overhead. V3 deletion vectors (Puffin-based, the same mechanism Delta Lake has used) make point deletes a metadata operation. The compaction overhead from delete files largely disappears, and the small-files-from-deletes problem this essay implicitly assumes is materially reduced.
• CDC and incremental processing from Iceberg. V3 row-level lineage (row IDs plus last-updated tracking) enables change-data-capture-from-Iceberg without external metadata. Some maintenance jobs that currently use snapshot-diffing patterns to figure out what changed can be simplified or removed entirely.

The bulk of this essay (Netflix's metadata-architecture lessons, Insider's lifecycle tiering, the petabyte-scale operational schedule, Z-ordering economics) is unaffected by V3. The "V3 is upcoming" framing from earlier drafts is wrong, because V3 shipped in 2025 and V4 is what's coming next.

Compaction is the operation that takes thousands of small Parquet files and rewrites them as a smaller number of large ones, and the reason it matters is mechanical rather than philosophical. Streaming writes (Kafka feeding Iceberg through Flink, for example) commit small batches frequently, so at petabyte-scale ingestion with a 100 MB target file size, you produce twenty thousand-plus files per table per day. Each S3 LIST operation has a roughly 5-millisecond baseline latency, so query planning across a million files turns into a thousand LIST calls, which is five seconds of overhead before the query starts. Parquet file open overhead adds another 50 milliseconds per file, which means ten thousand files becomes eight minutes of file opening before anything useful happens.

The fix is to rewrite small files into larger ones on a schedule. Iceberg exposes this as a system procedure you call from Spark SQL:

CALL system.rewrite_data_files(
  table => 'security_events.firewall_logs',
  strategy => 'binpack',
  options => map(
    'target-file-size-bytes', '512000000',  -- 512 MB target
    'min-input-files', '5',                 -- only compact if 5+ small files
    'partial-progress.enabled', 'true'      -- commit incrementally
  )
);

The "binpack" strategy is the simplest: group small files together until the target size is reached. The "partial-progress.enabled" option matters for long-running compactions. Without it, a failure mid-compaction loses all progress. With it, work commits in increments.

Frequency by table size

The right cadence depends on how fast each table grows. Three rough tiers:

• High-volume tables (500+ GB/day): daily compaction.
• Medium-volume tables (50–500 GB/day): weekly compaction.
• Low-volume tables (under 50 GB/day): monthly or on-demand.

A representative petabyte-scale schedule looks like this: compaction runs early in the daily maintenance window against the prior day's partitions on a day-hour-partitioned table, taking roughly 45 to 90 minutes for a couple of terabytes of input and costing on the order of tens of dollars a day on spot compute. Query latency on long (90-day) scans recovers substantially once the schedule holds, and I measured this part directly rather than leaving it directional: compacting a fragmented security table, 500 small files down to four, recovered hunting-scan latency by 3.2 to 5.3x from the file-count reduction alone, and up to roughly 10x when the rewrite was sorted on the query's range key like time, because the sort adds row-group pruning on top of the fewer-files effect — single host, Tier B, answers verified identical before and after, so read the file-count figure as a conservative floor. An independent micro-benchmark on a different dataset lands in the same shape, with metadata-only count queries barely moving and the data-touching scans recovering 50 to 66%.

One thing worth knowing is that the compaction cost-benefit ratio is asymmetric, because the compaction job costs tens of dollars per day while missing it costs most analysts on the team minutes per query for months, so it's worth running even when you're not certain it's needed.

Orphan files are the debris from writes that failed before committing. The sequence is mechanical: Spark writes new Parquet files to S3, the job fails before updating Iceberg metadata, the files remain in S3 but no snapshot references them, and S3 lifecycle policies don't know to delete them because they look like normal data.

Iceberg's "remove_orphan_files" procedure walks the table's data prefix, compares against referenced files, and removes the unreferenced ones. The "older_than" parameter is critical: set it conservatively to avoid deleting files from writes that are still in progress:

CALL system.remove_orphan_files(
  table => 'security_events.firewall_logs',
  older_than => TIMESTAMP '2025-11-28 00:00:00',  -- 48 hours ago
  dry_run => true  -- preview before deletion
);

The "dry_run" flag is worth using on every first execution. The procedure returns the list of files it would delete; eyeball it before turning off dry-run mode. A 48-to-72-hour safety buffer prevents the procedure from deleting files belonging to in-progress multi-hour compliance jobs.

A representative cadence: orphan cleanup runs monthly, in a quiet weekend window, with a 72-hour buffer. The reclaimed storage is small in percentage terms, well under 1% of the total, but it adds up across dozens of tables to a modest monthly S3 saving, which isn't large but is free money once the job is automated.

Netflix's pre-Iceberg logging system ran on Apache Hive and hit a wall at petabyte scale. The symptoms were specific. Listing partitions for query planning took minutes, not milliseconds. Schema evolution required full table rewrites; adding a column meant reprocessing petabytes, which is a weeks-long operation. There were no ACID guarantees during concurrent writes from the dozens of pipelines feeding the system.

Iceberg, developed at Netflix between 2018 and 2020 under Ryan Blue's leadership, moved partition information out of the Hive metastore and into Parquet-based manifest files stored alongside the data. The architectural shift produced three operational changes that matter for security architects sizing their own systems: listing partitions became a manifest-file read (milliseconds rather than minutes), schema evolution became a metadata-only operation (seconds rather than weeks), and concurrent writes from many pipelines became safe (ACID by construction).

Production validation: petabyte-scale tables, millions of partitions without performance degradation. The lesson for security teams ingesting 200 GB/day to 10 TB/day: you will not hit the metadata limits that Netflix's petabyte-scale logging system exercises. The architecture has 50-to-500x headroom for typical security data volumes.

On Hive-style systems, analysts have to know the partitioning scheme to write efficient queries. The query looks like this:

-- analyst must specify partition filters manually
SELECT * FROM security_events.firewall_logs
WHERE partition_date = '2025-11-30'
  AND partition_hour >= 10
  AND event_time > now() - interval '24 hours';

The analyst is reasonable to ask why partition_date and event_time are both there. They look redundant. The redundancy is structural (the partition columns are how the engine prunes data; the event_time is the actual semantic filter), but it's confusing and easy to forget. When the analyst forgets the partition filter, the query degrades into a full table scan, which can be tens of times slower.

Iceberg's hidden partitioning makes the partition filter implicit:

-- Iceberg applies partition filters automatically
SELECT * FROM security_events.firewall_logs
WHERE event_time > now() - interval '24 hours';

The engine sees the event_time filter, knows the table is partitioned by hours(event_time), and applies the appropriate partition pruning without the analyst writing it. Query plans show the pruning happening; the analyst doesn't have to think about it.

This is a well-documented Hive-to-Iceberg pattern: on partition-unaware Hive layouts a meaningful share of analyst queries miss the partition filter and run an order of magnitude or more slower than they should, and hidden partitioning closes that gap because the engine derives the partition predicate from the query rather than relying on the analyst to name the partition column, which is what lets the analyst experience keep scaling as the table grows.

Query latency degrading with no change in data volume

Symptom: a query that took 15 seconds in month one takes 90 seconds in month four, even though data volume and query patterns haven't shifted, which is almost always small-file proliferation. Confirm with a quick query against the table's files metadata:

SELECT
  COUNT(*) as file_count,
  AVG(file_size_in_bytes) / 1024 / 1024 as avg_size_mb
FROM polaris.security_events.firewall_logs.files
WHERE partition.event_date >= current_date() - interval '30' days;

If average size is under 128 MB and file count is above 5,000, run compaction immediately, then figure out why the scheduled compaction job stopped firing. Add the file-size alert so the next time it happens, you find out before the analysts do.

Catalog operations slow, time travel queries timing out

Symptom: listing snapshots takes 30+ seconds, time travel queries fail with timeouts. Cause: snapshot expiration hasn't run for months and the snapshot count is in the tens of thousands. Run aggressive expiration retaining the most recent 100 snapshots, then schedule weekly expiration.

S3 costs climbing without data volume increase

Symptom: S3 storage costs up 15% month-over-month, ingestion volume flat. Cause: orphan files from failed Spark jobs accumulating. Run orphan cleanup in dry-run mode first to see how big the problem is. If the orphan count is in the thousands, you have a Spark reliability issue worth investigating in parallel with the cleanup.

Partition strategy outgrown by data volume

Symptom: queries that used to scan a partition cleanly now hit out-of-memory errors. Cause: you started at 50 GB/day (daily partitioning was fine) and you're now at 800 GB/day (daily partitions are too large). Fix: add an hourly partition transform without rewriting existing data, using the partition evolution pattern from earlier in this essay. Hidden partitioning means analyst queries don't change.

AWS announced Amazon S3 Tables in December 2024 as a managed Iceberg service with automatic compaction, snapshot management, and a built-in Iceberg REST Catalog. Tables are queryable from Athena, EMR, Redshift, Glue, and third-party engines. For security teams that don't want to operate self-managed Spark compaction jobs, this is the operationally simpler option.

The trade-offs are real. You lose fine-grained control over compaction scheduling and target file sizes. You're locked into AWS-managed maintenance behavior, which may not match what your workload needs. And pricing is a separate consideration; the managed service adds cost on top of the underlying S3 storage. (Tier D note on the AWS pricing model: it's a vendor-controlled variable, and I'd verify current pricing rather than relying on launch-date numbers.)

My take: for new deployments under 100 TB/day where the operational complexity of self-managed Spark compaction isn't justified, S3 Tables is a reasonable starting point. For larger deployments, or for teams that need control over partition strategies that don't match AWS defaults, the self-managed pattern in this essay is still the right answer.