Format and compliance

Deletion vectors and GDPR: Iceberg's right-to-erasure story, honestly.

Until Iceberg V3, "Iceberg has a right-to-erasure problem" was a defensible architectural objection from compliance teams. V3 introduces Puffin-based deletion vectors (the same mechanism Delta Lake has used for years), which materially changes the technical conversation, but it does not on its own satisfy GDPR Article 17, because the technical mechanism is necessary without being sufficient.

Reading time: about 16 minutes. Evidence tier: B for the V3 deletion-vector spec mechanics (Apache Iceberg V3 specification, Puffin file format documentation). The regulatory claims below are my reading of GDPR, PCI-DSS, and HIPAA text and publicly available regulator guidance, and they are not legal advice. For a binding interpretation in your jurisdiction, consult named counsel and your data protection officer.

It's worth being precise about the V2 mechanics, because the compliance objection is about the specific gap between V2 and V3 rather than about Iceberg in general.

In Iceberg V2, a query like DELETE FROM users WHERE user_id = 'alice@example.com' did the following:

• Identified affected files. The engine scanned the table metadata to find data files that might contain rows matching the predicate.
• Wrote one or more delete files. Either a position-delete file (storing the row positions to delete within each affected data file) or an equality-delete file (storing the equality predicate for matching at read time).
• Recorded the delete files in a new snapshot. Subsequent reads of the table would consult the delete files and skip the matched rows.

After the operation, three things were true at once: queries against the table no longer returned the deleted rows, the original data files in object storage still contained those rows, and a future compaction operation would eventually rewrite the data files without the deleted rows, at which point the bytes were physically gone, but only when compaction ran.

The compliance objection lived in the gap between the first and third items. GDPR Article 17 requires that personal data be erased "without undue delay." Different EU data protection authorities have interpreted "without undue delay" in different ways. Some guidance points toward 30 days, some toward 72 hours, and the European Data Protection Board has not pinned this down with a single binding number in primary guidance I can cite. What is uncontroversial: a weekly compaction cadence on a high-volume table is almost certainly outside any reasonable reading of "without undue delay," and weekly was the typical V2 cadence for big tables because of the operational cost.

The V2-era workarounds were three:

• Run compaction on demand after every DELETE. Operationally expensive, especially for tables with frequent erasure requests, and the per-erasure cost was hard to amortize.
• Use crypto-shredding. Encrypt personal data with a per-subject key, then destroy the key when the subject requests erasure. The bytes remain but are computationally unrecoverable. Some regulator guidance accepts this as erasure; others have been more skeptical, particularly in EU contexts where "anonymization" and "pseudonymization" are sharply distinguished. Treat this as a legal-counsel question in your jurisdiction, not a software guarantee.
• Choose Delta Lake instead. Delta has had Puffin-based deletion vectors longer, with a faster compaction story. This was the V2-era recommendation for compliance-first deployments and the primary reason in my earlier Iceberg-vs-Delta framework to prefer Delta when erasure obligations were a primary requirement.

V3 makes the third workaround unnecessary by closing the capability gap inside the primary table. The first two workarounds (on-demand compaction, crypto-shredding) remain available and may still make sense for specific patterns, but they're no longer architectural escape hatches for an Iceberg-specific weakness.

There are three categories of erasure problem that V3 deletion vectors do not solve, and I've ordered them by how often they catch teams off guard.

1. Time travel and snapshot retention

Iceberg's time-travel feature (querying the table as it existed at a previous snapshot) is structurally incompatible with hard erasure. If you can query "the table as it existed yesterday," then yesterday's snapshot still contains the personal data of a subject who requested erasure today, and V3 doesn't change that.

To honor an erasure request fully, you have to also expire snapshots that contain the deleted data, which means your maximum time-travel window is bounded by your erasure-policy SLA. For a 30-day treatment of "without undue delay," your maximum time-travel retention is 30 days. For a 72-hour treatment, three days. There is no architectural escape from this tradeoff; it's a policy choice between operational recovery ("we can roll the table back if an ETL job corrupts it") and erasure fidelity ("we honor deletion requests across all readable state").

Most production deployments split the difference: a bounded time-travel window aligned to the erasure SLA, and a documented guarantee that snapshots older than that window no longer contain rows subject to deletion requests. This is a tradeoff Iceberg shares with Delta Lake and every other versioned table format, so it isn't an Iceberg-specific weakness.

2. Cold-tier archives

If you've tiered old Iceberg data to S3 Glacier Deep Archive, Azure Archive, or physical tape backups (a common pattern for long-retention security logs), deletion vectors don't reach those archives. The cold-tier copy of the data file still contains the deleted rows, and the Puffin metadata that records the deletion lives only in the warm tier.

The V3 fix for this is "rewrite the cold-tier file too," but cold-tier rewrites are expensive (retrieval fees, restore latency, tape rewrites) and operationally awkward. The pattern most teams use in practice is to not tier rows to cold storage until they are past the erasure window, which means cold-tier data is, by policy, no longer subject to right-to-erasure requests because the retention period has expired.

That policy is legally defensible if and only if the cold-tier retention horizon is itself within whatever you've told regulators and data subjects. If you're keeping cold-tier copies indefinitely and treating them as exempt from erasure, that's a conversation to have with counsel rather than a configuration choice you can settle on your own.

3. Replicas, backups, and downstream copies

Disaster-recovery replicas, cross-region copies, third-party backup tools (Veeam, Druva, NetBackup, Cohesity), and downstream warehouses fed by CDC pipelines: none of these read Iceberg metadata or deletion vectors. They see Parquet files as opaque blobs. An erasure request that propagates through V3 deletion vectors to the primary table does not propagate to the replicas, backups, or downstream copies on its own.

This is, in my experience, the most underestimated piece of the right-to-erasure architecture problem. The V2-era teams that chose Delta Lake "for GDPR compliance" had this exact issue and usually solved it the same way: maintain a cross-system erasure orchestrator that propagates deletion requests to every copy of the data, with an auditable log of which systems acknowledged the request. V3 deletion vectors don't change the orchestration problem; they just make the primary table compliant on its own.

The legal posture I'd want before defending an Article 17 response is roughly: "the row was marked deleted in the primary table at time T1, physically rewritten by T2, propagated to replicas by T3, purged from backup catalogs by T4, and confirmed removed from the downstream warehouse by T5, all within the SLA we committed to." V3 helps with T1 and T2. The rest is orchestration, contract terms with backup vendors, and downstream-pipeline design. None of it is implied by the bullet "Iceberg supports deletion vectors."

If you're operating a security data lake on Iceberg V2 with an active right-to-erasure obligation under GDPR, CCPA, PCI-DSS, or HIPAA, V3 should be on your near-term roadmap. Five specific moves I'd prioritize, in order:

• Audit your current erasure SLA end-to-end. What's the lag between an erasure request and the bytes being physically gone in every copy of the data: primary, replicas, backups, downstream warehouses, archives? For most V2 deployments running weekly compaction on the primary, the primary-table lag is days-to-weeks; the full-stack lag is usually worse. Compare to what you've committed to data subjects and regulators. If there's a gap, that's the size of the fix you need to scope.
• Plan the V3 engine upgrade. Verify that your query engines (Spark, Trino, DuckDB, Snowflake, ClickHouse) support V3 deletion vectors and the Puffin file format at the versions you run. As of mid-2026, Spark 4.1 has full support, Trino's V3 support is rolling out, DuckDB has partial support depending on the build, and Snowflake's adoption is catalog-side and ongoing. Pin this to your engine roadmap, not to the abstract Iceberg version.
• Update your compaction schedule. Once on V3 with engine support, switch to deletion-vector-aware selective compaction running hourly or continuously. The cost increase over weekly full-table compaction is modest because the metadata overhead is small; the compliance benefit on the primary-table side may be substantial.
• Don't extend your time-travel retention. The time-travel-versus-erasure tradeoff is unchanged. Continue to bound time-travel retention by your erasure SLA, and document the tradeoff explicitly so internal stakeholders understand why the operational-recovery window is capped.
• Audit the rest of the data estate. Replicas, third-party backups, downstream warehouses, machine-learning training sets, analytics caches, BI extracts. Build or buy a cross-system erasure orchestrator. This is the part most easily neglected when the primary-table story looks clean.

If you're choosing between Iceberg and Delta Lake for a new deployment with compliance as a primary requirement, the V2-era recommendation "choose Delta for GDPR" is no longer the deciding table-format argument. The two formats use convergent mechanisms for the primary table. The remaining decision factors (vendor neutrality, multi-engine support, ecosystem maturity, Databricks commitment, the longer Delta production track record on these specific erasure features) are the ones that should drive the choice. Compliance still matters; it just no longer narrows the choice on its own.

Five moves I'd recommend to a security architect operating a lakehouse under any active erasure regime, roughly ordered by cost to execute:

• Write down the SLA you've already committed to. Pull the privacy notice, the data processing agreement template, the BAA template if you're in healthcare, and whatever the contract says about response time to deletion requests. The committed number is the design constraint. A lot of teams design to "best effort" because they've never written the number down.
• Map the actual copies of the data. Primary table, all snapshots, all replicas (DR, cross-region, read replicas), all backups (in-platform and third-party tooling), all downstream consumers (warehouses, BI tools, ML training pipelines, caches), all archives. This is usually the longest list in the room and the one that drives the rest of the architecture.
• Decide where deletion vectors actually help. V3 deletion vectors solve the primary table cleanly. They do nothing for the rest of the copy list. Cost the V3 upgrade against the piece of the problem it actually addresses, not against the regulatory obligation as a whole.
• Invest in the orchestration layer. Whatever you call it (privacy-rights service, deletion orchestrator, subject-access fulfillment), this is the load-bearing piece. A well-instrumented orchestrator with named retry, named acknowledgment from every system, and an auditable per-request log is more important to your regulatory posture than any specific table format choice.
• Get named counsel on the interpretation questions. Cryptographic erasure versus physical erasure. "Without undue delay" in your jurisdictions. Whether retention exceptions apply to cold-tier archives. Whether QSA opinion accepts deletion-vector compaction as "secure deletion." These are not technical questions, and resolving them with a vendor blog post is how teams end up with surprises at audit time.

None of this is meant to dim the V3 story, because Iceberg V3 deletion vectors are a real, useful, and overdue improvement, and the right-to-erasure conversation around Iceberg-based security lakehouses is materially better now than it was in 2024 now that the architectural objection that drove the choice for the last decade no longer holds. That's a shift worth marking, but it's not the same shift as "Iceberg is now GDPR-compliant," and I'd rather draw the line clearly than leave architects holding a regulatory exposure they didn't realize was still there.