Technology deep-dive

Meta-catalogs and asset context in federated environments.

A federated security architecture has two visibility problems that look related but live at different layers. Where does the data physically live across multiple data catalogs (Hive, Polaris, Unity, PostgreSQL)? And what does any given asset in that data actually mean to the organization (who owns it, how critical is it, where is it supposed to be)? Meta-catalogs answer the first question. CAASM (Cyber Asset Attack Surface Management) answers the second. Together they're how a federated SOC gets cross-organization visibility without flattening every business unit onto the same platform.

Reading time: about 17 minutes. Evidence tier: B overall (vendor documentation from Apache Gravitino, Apache Polaris, and Databricks Unity Catalog; Axonius customer case studies; practitioner reports). Gravitino-specific claims are flagged as early-maturity throughout: the project is in Apache incubation and production references are still emerging.

These terms get used loosely in vendor marketing, but the distinction matters once you're choosing components.

A unified catalog stores and manages metadata for one format or one ecosystem. Apache Polaris is a pure Iceberg catalog: vendor-neutral, Apache Software Foundation governance, REST API. Databricks Unity Catalog is multi-format (Delta Lake natively, plus Iceberg and Hudi) but Databricks-centric in deployment and lifecycle. The Hive metastore is a unified catalog for the legacy Hadoop ecosystem and still widely deployed.

A meta-catalog doesn't store table metadata itself. It federates access to multiple underlying catalogs and exposes a single API that knows how to route a query to the right one. Apache Gravitino is the most prominent open-source example. The mental model is:

Gravitino API
   |-- Polaris Catalog (Iceberg tables)
   |-- Unity Catalog (Delta Lake tables)
   |-- Hive Metastore (legacy Hadoop tables)
   |-- PostgreSQL (relational data)

Gravitino is a metadata router rather than a data proxy, so the data still flows directly from storage (S3, ADLS, GCS) to the query engine (Trino, Spark, Dremio) while Gravitino only routes the metadata lookup. Governance is delegated to the underlying catalogs, which means Gravitino checks catalog-level role-based access control, the underlying catalog enforces table-level permissions, and the query engine applies row filters or column masks where the catalog supports them.

A practical example. Imagine a detection engineer wants to correlate CloudTrail events in Polaris-backed Iceberg, user records in PostgreSQL, and legacy Zeek connection logs in Hive, all in one query through Trino:

-- Join Iceberg (Polaris), PostgreSQL, and Hive via Gravitino
SELECT
    c.eventName,
    c.userIdentity.principalId AS user_id,
    u.email,
    u.department,
    z.src_ip,
    z.dest_ip
FROM gravitino.polaris_catalog.cloudtrail AS c
JOIN gravitino.postgres_catalog.users AS u
    ON c.userIdentity.principalId = u.user_id
LEFT JOIN gravitino.hive_catalog.zeek_conn AS z
    ON c.sourceIPAddress = z.src_ip
WHERE c.eventTime > NOW() - INTERVAL '1' DAY;

Trino talks to Gravitino. Gravitino routes the metadata lookups for each table to Polaris, PostgreSQL, and Hive. The actual join happens in Trino's query engine, reading data directly from S3 and the underlying PostgreSQL instance.

Meta-catalogs earn their keep on organizational complexity more than on technical complexity, because the technical complexity (joining tables across heterogeneous catalogs) can usually be solved by dbt and patience, while the organizational complexity is what makes Gravitino worth deploying.

1. Multi-decade data platform evolution

A financial services SOC with fifteen years of data infrastructure. Roughly 1.5 PB in Hive (still queried for historical investigations), 800 TB in Delta Lake (active SOC analytics), 500 TB in Iceberg (the modern lakehouse), and 100 GB in PostgreSQL (threat intel, asset inventory, CMDB).

Without a meta-catalog, engineers write four queries (Hive via Beeline, Delta via Databricks SQL, Iceberg via Trino, PostgreSQL via psql) and then combine the results in a spreadsheet. The composite case studies I've seen put that work at two to four hours per cross-platform investigation. With Gravitino, the same investigation collapses to a single SQL query and lands in the fifteen-to-twenty minute range, so the savings show up in incident response time rather than in storage cost.

2. Federated conglomerate with independent business units

A global conglomerate with twelve acquired companies, each running independent security operations. BU-A on Databricks with Unity Catalog. BU-B on Snowflake. BU-C on Iceberg with Polaris. BU-D on Google BigQuery. When corporate security investigates a supply chain attack that touches three of those BUs, the choice without a meta-catalog is CSV exports and manual joins in a spreadsheet, measured in days. With a meta-catalog, it's a single query that joins all three sources in seconds. This is the scenario where Gravitino's value is least ambiguous, because the alternative is so painful.

3. Data migration with a dual-write window

A security team migrating from Hive to Iceberg over eighteen months. During the dual-write phase, detection engineers need to query both catalogs to validate that the migration preserved correctness. A simple consistency check looks like this:

-- Validate Hive -> Iceberg migration consistency
WITH hive_counts AS (
    SELECT COUNT(*) AS cnt
    FROM gravitino.hive_catalog.cloudtrail
    WHERE date = '2025-11-01'
),
iceberg_counts AS (
    SELECT COUNT(*) AS cnt
    FROM gravitino.polaris_catalog.cloudtrail
    WHERE eventTime::date = '2025-11-01'
)
SELECT
    h.cnt AS hive_row_count,
    i.cnt AS iceberg_row_count,
    ABS(h.cnt - i.cnt) AS discrepancy
FROM hive_counts h, iceberg_counts i;

Any discrepancy above zero indicates data loss in the migration. Gravitino enables this kind of systematic validation across a year of dual-written data without rewriting the validation harness for each catalog.

Meta-catalogs solve a structural problem, in that they tell the query engine where data lives across multiple catalogs, but they do not tell anyone what that data means in the context of the organization, and that gap is where CAASM (Cyber Asset Attack Surface Management) comes in.

The cleanest way to motivate CAASM is with an alert that does not need it, and then an alert that does. At 10:47 AM on a Tuesday, the SIEM fires. Sensitive S3 bucket accessed from an unusual IP. User alice@company.com. IP 203.0.113.45. Action s3:GetObject. Bucket finance-pii-customer-data. Severity HIGH.

Without asset context, the analyst's playbook is roughly: check Active Directory, check the CMDB for the IP (no results), query thirty days of CloudTrail, call Alice's manager (voicemail), get transferred through three teams, and resolve the alert as a false positive fifty minutes later. Alice's home office IP, approved work-from-home setup, never seen the alert subsystem before because nothing had ever joined her IP record against her bucket access pattern.

With CAASM enrichment, the same alert arrives with annotations attached. The user is Executive Assistant in the CFO Office. The IP owner is Alice, Home Office Router, approved work-from-home, first seen five months ago. The bucket is classified Critical, holding PII and Financial Data. The expected behavior baseline says Alice accesses this bucket daily from her home office. The investigation collapses to twelve seconds. The severity auto-adjusts from HIGH to INFO.

CAASM doesn't eliminate alerts here, but it does move the false-positive filtering out of the analyst's head and into the data, where the same logic can run across every alert instead of being re-derived by hand each time.

CAASM is useful in most organizations, but it becomes close to essential in a federated one, because federation introduces four asset problems that single-platform organizations don't have at the same severity.

Siloed CMDBs. BU-A uses ServiceNow, updated quarterly. BU-B keeps a spreadsheet, updated "when we remember." BU-C runs BMC Remedy. The central SOC alerts on device 10.15.23.87 and the owner-identification workflow turns into emails to fifteen IT teams and a three-day wait. With CAASM, the device is auto-discovered from BU-C's network scanners and Active Directory, and the owner is identified in twelve seconds, and the SOC value comes less from the discovery itself than from the cross-BU ownership graph that ties it together.

Inconsistent naming. BU-A calls a thing a "laptop," BU-B an "endpoint," BU-C a "workstation," BU-D a "mobile device." CAASM normalizes those to a canonical "Windows Device" so the SOC can write a single detection that doesn't need four OR-clauses.

Stale data. The quarterly CMDB update means a device that appeared on February 5 may not show up in the system of record until April 1, which leaves the security team blind to it for two months, whereas CAASM platforms run continuous discovery (typical refresh intervals are fifteen minutes to twenty-four hours, depending on the data source) so that gap closes from months to hours.

Shadow IT. BU-C spins up an AWS account for a proof-of-concept, never reports it to Central IT, runs fifty EC2 instances at $15K a month, and is invisible to the SOC until something goes wrong. CAASM that's been wired into AWS Organizations discovers the shadow account on its next sync.

The integration pattern I see working most reliably is daily export from the CAASM platform into an Iceberg table, enrichment via dbt at query time, and context-aware detection rules that join the enriched logs against the asset table.

Step 1: export CAASM inventory

Every major CAASM platform exposes an export API. The shape is roughly:

curl -X POST https://api.axonius.com/api/assets/export \
  -H "Authorization: Bearer $AXONIUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "json",
    "fields": ["asset_id", "asset_type", "owner", "department",
               "location", "ip_addresses", "mac_address",
               "criticality", "vulnerabilities", "last_seen"]
  }' > caasm_inventory.json

Daily cadence is sufficient for most workflows. Faster than daily (hourly, real-time) adds complexity with diminishing returns, because the CAASM platform's upstream sources usually don't refresh faster than once an hour anyway.

Step 2: dbt enrichment

Load the inventory into an Iceberg table and join security logs against it at query time. The dbt pattern looks like this:

-- models/enriched/ocsf_with_asset_context.sql
{{ config(materialized='incremental') }}

SELECT
    ocsf.time,
    ocsf.src_endpoint_ip,
    ocsf.activity_name,
    ocsf.actor_user_uid,
    -- CAASM enrichment
    caasm.asset_owner,
    caasm.asset_type,
    caasm.criticality,
    caasm.department,
    caasm.location,
    caasm.vulnerabilities,
    caasm.last_seen,
    -- Context flags
    CASE
        WHEN caasm.criticality = 'Critical'
             AND ocsf.activity_name IN ('DeleteBucket', 'ModifySecurityGroup')
            THEN 'high_risk_action_on_critical_asset'
        WHEN caasm.location != caasm.expected_location
            THEN 'asset_location_anomaly'
        WHEN caasm.last_seen < CURRENT_TIMESTAMP - INTERVAL '7 days'
            THEN 'stale_asset_activity'
        ELSE 'normal'
    END AS risk_flag
FROM {{ ref('cloudtrail_ocsf') }} AS ocsf
LEFT JOIN {{ ref('caasm_asset_inventory') }} AS caasm
    ON ocsf.src_endpoint_ip = ANY(caasm.ip_addresses)
{% if is_incremental() %}
WHERE ocsf.time > (SELECT MAX(time) FROM {{ this }})
{% endif %}

The performance characteristic that matters: the CAASM inventory is typically 10K to 100K rows. The security logs it joins against are 100M to 10B rows. This is the canonical broadcast-join pattern. Replicate the small CAASM table to every query node, stream the large log table past it. Query engines like Trino and Spark handle this automatically when the small side is below the broadcast threshold.

Step 3: context-aware detection rules

The payoff is detection rules that filter on context rather than on raw event signatures alone. A traditional, context-free rule:

SELECT * FROM cloudtrail_ocsf
WHERE activity_name = 'DeleteBucket'
  AND time > NOW() - INTERVAL '1 hour'

The composite case studies I work from suggest this kind of rule produces somewhere in the range of 40 to 50 alerts per hour in a mid-size enterprise, most of which are Alice deleting yesterday's backup bucket. The context-aware version:

SELECT
    time, src_endpoint_ip, activity_name,
    asset_owner, location AS current_location,
    expected_location, criticality
FROM enriched.ocsf_with_asset_context
WHERE activity_name = 'DeleteBucket'
    AND time > NOW() - INTERVAL '1 hour'
    AND (
        (criticality = 'Critical' AND location != expected_location)
        OR (last_seen < NOW() - INTERVAL '30 days')
        OR (vulnerabilities LIKE '%CRITICAL%')
    )
    AND NOT (asset_owner = 'alice@company.com'
             AND location = 'Home Office')

Vendors and practitioners cite false-positive reductions loosely in the 70% to 95% range for context-aware rewrites, though I haven't found a single rigorous published study behind those numbers, so treat the band as directional. My honest read: the upper end of that range is achievable for a handful of well-understood rules, and the typical result across a full detection library is closer to the middle of the range once you account for rules that don't have obvious context to add.

The two layers answer different questions. The meta-catalog answers where does the data physically live across our catalogs. CAASM answers what does any given asset in that data mean to the organization. A federated security architecture needs both because each one alone is incomplete.

A meta-catalog without CAASM lets a detection engineer write a single SQL query against four catalog types and still not know who owns the device the query returned. A CAASM platform without a meta-catalog produces a beautifully curated asset inventory that nobody can join against the four different log stores where the actual events live. The combination is what makes "we have visibility across the federation" a defensible statement rather than a marketing one.

For the rollout sequence (when to deploy which layer first, how to phase the budget, how to avoid the common ordering mistakes) see the federated rollout playbook, though the short version is that you stand up the primary catalog first (Polaris or Unity), wire CAASM into it as the first enrichment source, and only add Gravitino once you have three or more catalog types that need to be queried in a single statement on a daily basis.