Unity Catalog vs Polaris vs Nessie: choosing the catalog for security data.

The Iceberg catalog is more than a metadata registry, because it's also the enforcement point for who can read which tables, whether row-level security or column masking can be expressed at all, how schema changes are audited, and which query engines can participate without a custom connector. Pick the wrong catalog and the next year is spent writing authorization middleware, while picking the right one means the access-control questions a SOC has to answer collapse into table grants and namespace policies.

That's why I treat catalog as an Iceberg-pillar-adjacent decision. The table format gives you ACID transactions, schema evolution, and time travel, but it's the catalog that decides whether those features are governable, because without a catalog you have files in S3 whereas with one you have governed data assets and an answer to the auditor's question about who queried CloudTrail last Tuesday.

One qualifier I want to set down before the comparison, because it changes how much weight to put on the choice. In a general enterprise data platform the catalog can become the moat, the place that owns governance and therefore owns the customer. In security specifically I think that's mostly wrong. The governance security teams actually fight over is per-event: row-level ABAC on individual records, provenance on where a given event came from, and audit-retention that survives an investigation timeline. Those concerns push the real lock-in down to the engine and pipeline layer, where the policy is enforced and the data is shaped, not up to the catalog that registers tables. The counter-evidence is concrete: Iceberg V3 row lineage (Iceberg 1.9.0, April 2025) puts per-row provenance and last-updated tracking in the table format itself, readable by any engine that surfaces it and independent of which catalog wrote it. When the audit primitive lives in the open format, swapping Polaris for Nessie doesn't cost you the audit trail. So the catalog choice (Unity vs Polaris vs Nessie vs DuckLake) matters less for lock-in than the engine choice does. It still matters a great deal for operational fit, which is what the rest of this piece is about.

I'd been asserting that swap so I ran it. On a single host I wrote and read the same OCSF table through three independent catalog implementations over one MinIO store: the iceberg-rest reference fixture (the Java reference implementation), Nessie (Java and Quarkus, with the Git-style branching above), and Lakekeeper (a Rust catalog backed by Postgres). The query returned the identical answer, rdp=125, through all three, with the data never moving and only the catalog changing underneath it (./moar swap-catalog in the MOAR reference stack, 2026-06-07). I ran the same check one layer down with ./moar swap-format, and the same OCSF batch returned the identical answer across Iceberg and DuckLake on the same store, so the table format is replaceable on the same footing as the catalog. Three separate codebases agreeing on the answer is a portability signal rather than a single-implementation coincidence, which is the central claim of this piece demonstrated rather than asserted. I want to keep the framing honest about what it is: an answer-equality check on one host, not a production migration or a performance result, and Unity Catalog isn't in that stack, so its behavior here stays documented rather than first-party.

For background on why Iceberg over Delta in the first place, see Iceberg vs Delta Lake for security data. This piece picks up where that one ends, with the table format chosen and the catalog now on the table.

An Iceberg catalog tracks four things that the table format itself doesn't pin down: which S3 paths hold which tables, the version history of schema changes, the snapshot list that enables time travel queries, and how data is physically partitioned, which is the registry function and table-stakes for any of the options here.

For security teams, the catalog is also the enforcement surface for governance, which is where it decides who can query the CloudTrail tables, whether the source IP column is visible to a particular role, what gets written to the audit log when a user runs a query, how OCSF normalized tables stay consistent when fifteen of them have to update atomically, and which dashboards depend on which raw log tables so that a schema change doesn't silently break a detection rule.

Not every catalog does every one of these, though, because some assume a single cloud, some require specific query engines, and some give you row-level filters while others stop at table-level grants. The trade-offs are real, and they're the reason this decision deserves more than a Slack-thread skim.

Snowflake donated Polaris to the Apache Software Foundation in 2024 as the vendor-neutral counterweight to Unity Catalog. The design principles are deliberately narrow: pure Iceberg (no Delta, no Hudi), standard Iceberg REST catalog protocol, cloud-agnostic deployment, and explicit support for any Iceberg-compatible query engine (Spark, Trino, Dremio, StarRocks, ClickHouse). Snowflake's own Iceberg Tables product uses a Polaris-compatible API, and customers can self-host Polaris or use the Snowflake-managed service.

Polaris's access model is table-level and namespace-level RBAC. You grant SELECT on cloudtrail_logs to a security_analysts role, grant ALL PRIVILEGES on detection_rules to soc_lead, and revoke when an analyst leaves. Namespace-level grants are useful for multi-cloud splits. Give cloud_security_team access to everything in the aws_logs namespace, give azure_security_team access to everything in azure_logs. The privilege model is granular at the operation level (READ, WRITE, CREATE_TABLE, DROP_TABLE, MANAGE_CONTENT) and supports principal-based authentication with service principals and OAuth or OIDC identity providers.

What Polaris does not have is row-level filters and column masking, so if you need those, you implement them at the query engine layer (Trino row filters, Dremio column policies) or via dynamic views, and you accept that the policy now lives in two places instead of one. For a single-tenant security platform on isolated infrastructure that's an acceptable trade, though for a multi-tenant MSSP it isn't.

Audit logging in Polaris is catalog-level (who accessed which tables, when) and integrates with CloudWatch and Azure Monitor. Query-level audit is delegated to the engine, which means a complete audit trail requires correlating Polaris logs with engine logs. For SOC 2 or ISO 27001 the catalog-level trail is usually enough, but for GDPR right-to-access requests that need column-level lineage it isn't.

The strategic read on Polaris is that Snowflake is using it the way Databricks used Spark a decade ago: donate the core, build commercial value on top, and let the open standard pull the ecosystem along. That dynamic is not by itself a reason to choose or avoid Polaris, but it shapes how the roadmap is likely to evolve. I'd hedge on long-horizon claims about Polaris's feature trajectory; the catalog is Apache-governed but the practical roadmap pressure comes from Snowflake's commercial priorities.

DuckLake is the 2026 entrant I'm watching closely, and the framing I've landed on is less "separate wildcard" than "different point on a spectrum." Catalog metadata can be realized three ways behind one Iceberg read contract: as static metadata files (the Unity, Polaris, and Nessie default), as metadata held in a SQL database, or as a virtual layer synthesized on demand (Streambased's ISK is the early articulation of that virtual end). DuckLake is the database-backed point. Instead of a catalog service sitting beside the table format, it keeps catalog metadata in a single SQL database queried directly by DuckDB. The early benchmark claims are striking. DuckDB Labs has cited two-orders-of-magnitude improvements on streaming workloads compared to file-based catalog metadata. That's the kind of number that deserves scrutiny, and I haven't independently benchmarked it against OCSF-shaped security data.

There's a caveat that decides where DuckLake fits, and it carries most of the weight here. DuckLake reaches outside engines through DuckLake-native clients rather than through an Iceberg REST endpoint, and its published roadmap through v2.0 doesn't add one. So a Trino, Spark, or ClickHouse process can't simply point an Iceberg REST connector at DuckLake the way it can at Polaris. Cross-engine consumption is copy-bridged: you replicate metadata out to an Iceberg-REST-fronted tier rather than reading DuckLake in place. That places DuckLake squarely in the warm, operational tier, fast single-engine access close to ingest, with an Iceberg-native catalog still doing the cross-engine retention work behind it.

The strategic question DuckLake raises: if catalog metadata can live in a SQL database queried alongside the data, do you need a dedicated catalog service for the operational tier at all? For small to mid-scale security deployments where the catalog itself is not a scaling bottleneck, DuckLake may simplify the warm-tier stack enough to compete on operational footprint rather than feature parity. For petabyte-scale deployments running concurrent Trino, Spark, and ClickHouse workloads against the same tables, the dedicated Iceberg-REST catalog service is probably still the right answer in 2026, precisely because DuckLake can't yet serve those engines without the copy.

I'd flag this as a watch-don't-bet item. DuckLake is early. The DuckDB ecosystem is real and the performance work is credible, but a category-redefining shift takes more than a v1.0 release. For architects making catalog decisions in 2026, the prudent move is to design the catalog layer so that a future DuckLake operational tier is possible without rewriting governance, which is one more argument for Iceberg-native, vendor-neutral choices over deep Unity Catalog coupling. For the deeper treatment of the metadata-realization spectrum and the format-war stakes, see Iceberg V4 vs DuckLake.

One thing the catalog comparison can obscure: regardless of which catalog you pick, Iceberg tables require periodic maintenance to keep query performance and storage costs under control. Catalogs track snapshots; they don't compact small files, expire old snapshots, or clean up orphans. The one exception is Unity Catalog managed tables on Databricks, which run maintenance automatically. Everywhere else, the maintenance job is yours to schedule.

Three operations have to be running somewhere. File compaction: streaming ingestion produces thousands of small Parquet files that slow scans; daily or weekly compaction into 1-2 GB files restores performance. Snapshot expiration: Iceberg keeps snapshots indefinitely, which bloats storage; weekly expiration beyond your retention window (often 90 days for compliance, 7 days for ops tables) keeps metadata reasonable. Orphan file cleanup: failed writes leave Parquet files in S3 that no snapshot references; monthly cleanup typically recovers 2-5% of total table size.

-- Daily compaction target 1 GB files
CALL spark_catalog.system.rewrite_data_files(
  table => 'security_logs.cloudtrail',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '1073741824')
);

-- Weekly snapshot expiration past 90-day window
CALL spark_catalog.system.expire_snapshots(
  table => 'security_logs.cloudtrail',
  older_than => DATE_SUB(CURRENT_DATE(), 90),
  retain_last => 10
);

-- Monthly orphan file cleanup
CALL spark_catalog.system.remove_orphan_files(
  table => 'security_logs.cloudtrail',
  older_than => TIMESTAMP '2026-04-01 00:00:00'
);

The catalog-specific story breaks down like this. Unity Catalog managed tables on Databricks automate all three operations via TBLPROPERTIES. Polaris delegates maintenance to your query engine: schedule Spark jobs via Airflow, dbt, or Kubernetes CronJob. Nessie does the same, with the added wrinkle that you maintain each branch independently, though Nessie's multi-table transactions let you compact across all fifteen OCSF tables atomically.

Operational cost is modest, roughly 30-60 minutes per week for ten to twenty Iceberg tables once the automation is wired up, and the performance return is worth it because compaction can improve query speed 2-10x when fewer files means faster metadata reads and better predicate pushdown. Catalogs that automate this for you are buying back engineer time at the cost of vendor coupling.

For the architectures I most often work with (dedicated security data plane, single tenant, table-level RBAC sufficient, multi-engine query stack) Apache Polaris is the default, because vendor neutrality preserves the exit path, Iceberg REST is a real standard, table-level access control plus network isolation covers the governance requirement, and maintenance automation via Spark plus Airflow is a known pattern.

I'd choose Unity Catalog over Polaris in two specific cases. First, organizations already on Databricks where the integration benefits compound, where Unity is strictly additive rather than a new vendor decision. Second, security platforms that genuinely need catalog-enforced row filters or column masks: multi-tenant MSSPs, GDPR-driven column-level lineage, federated regional teams with legally restricted access patterns.

I'd choose Nessie when the data engineering team treats detection logic as code: version control, feature branches, atomic rollback. That's a real shape, more common in mature security data engineering teams than in average SOCs, and the Git mental model pays off there in a way it doesn't elsewhere.

The wrong answer is running no catalog at all, because an Iceberg deployment without governance is a data swamp with extra steps, so pick one of the three, wire up the audit trail, and move on to the next decision.