OCSF and AI parsers
The harness, not the model
The question an AI-generated parser raises is whether it can be trusted, and the failure I care about is a mapping that looks right, ships, and writes a field to an OCSF path nobody queries. A lab test across a local-to-frontier model ladder, a deterministic mapper that can only emit real paths and a frontier model re-scored behind the same schema validator shows that the validator supplies the safety, the model supplies the accuracy and neither catches a mapping that is valid and wrong, which leaves a set of questions to ask any product that sells AI-generated parsers.
Reading time: about 18 minutes. Evidence tier: B for my lab runs (the mapping oracle and the deterministic mapper, June 2026, plus a post-hoc re-score on 2026-11-16); D where labelled, for one single-reader adjudication; C for one vendor phrase, with the bias flagged; A for the OCSF 1.8.0 schema itself.

The failure
A rule that never fired
The worst detection failure I've watched, and I've written it up in The ground you're already standing on, was a rule for a specific kind of authentication event that passed review, deployed clean and matched nothing for months, because somewhere upstream in the integration the field it keyed on had been mapped to a path that didn't exist in the data it actually ran against, so nothing in the pipeline raised an error and the search simply returned zero every time it ran, and all of that came from one mapping, which is why the question an AI-generated parser raises, whether it can be trusted, comes down for me to that same failure produced at machine speed, a mapping that looks right, validates against nothing in particular, ships to the lake and writes a field into a column nobody queries.
The short version of what my lab found is that a validator built from the full, version-pinned OCSF schema takes invented paths to zero for any model, by construction, that behind it a frontier model keeps nearly all of its accuracy, 57 to 66% of paths exactly right against about 16% for a crude deterministic alias table, and that neither the validator nor the model catches a mapping that is valid and wrong, which I think changes how a team should evaluate any product that sells AI-generated parsers.
This is the evidence companion to LLM-assisted OCSF mapping, which covers the mapping tax across twelve projects I time-tracked along with the grounding-control result (a correct note about an OCSF class did no better than a note about the wrong class), and to the one-paragraph summary of the deterministic mapper on the lab page, and what it adds is the full model ladder with its run-to-run variance, the frontier model behind a schema validator, what a validator built from a schema subset did to the frontier numbers, and a count of how often a valid mapping is simply wrong.
The test
The test and the failure it measures
The task is the narrowest useful slice of parser work: given one source field and the target OCSF class, name the OCSF 1.8.0 attribute path it belongs at. The answer key is 141 real-path mappings across six vendor sources (Okta System Log, CrowdStrike Detection Summary, Palo Alto PAN-OS TRAFFIC, Cisco ASA connection syslog, Cisco Umbrella DNS and Zscaler ZIA Web), hand-curated from vendor documentation and the OCSF schema and never from a model (ocsf-mapping-oracle/README.md, Tier B). In June every predicted path was checked against the closure of a checked-in, transcribed subset of the OCSF 1.8.0 schema (ocsf-mapping-fidelity/schemas/ocsf/PROVENANCE.md), and that subset matters for the frontier numbers below.
The primary metric is what I call the silent-error rate, the share of fields mapped to an OCSF path that does not exist in 1.8.0. It is the security-relevant failure because a pipeline that does not validate paths will happily write src_endpoint.hostname_fqdn into a column nobody queries, and every detection that reads the real attribute will see nothing and say nothing. I also report exact-match path-correctness against the gold, with the caveat that exact match penalizes defensible alternatives, because OCSF often admits more than one reasonable home for a field.
The five grounding conditions were no help at all (the field and class only), a partial list of the class's valid paths in the prompt, that list plus a conceptual note on what actor, user, endpoint and traffic mean for the class, that list plus a reasoning discipline (decide actor versus object first, choose the most specific path, never invent), and a control that hands the model a grounding note for the wrong class, which is how I tell content apart from extra tokens. The partial list was the first 80 of the class's paths in alphabetical order, none more than one level deep, which for the network classes means everything from action to device.* and nothing under src_endpoint, dst_endpoint or traffic, and only 17 of the 141 gold paths appeared in the list their prompt showed.
The ladder
What the models did
Across the ladder, the silent-error rates looked like this (lab, June 2026, Tier B, single-shot, temperature 0 for the local models; ocsf-mapping-oracle/results/RESULTS.md). The local rows are as the June scorer counted them, and re-scoring against the full schema moved them by at most a point where cached predictions existed to check; the frontier row is run one re-scored against the full OCSF 1.8.0 schema, because the June scorer, built on a schema subset, roughly doubled it (next section).
| Model | No grounding | Partial path list | List + concept note | List + discipline | Wrong-class note |
|---|---|---|---|---|---|
| phi3 (3.8B, local) | 0.99 | 0.69 | 0.72 | 0.60 | 0.72 |
| gemma4:26b (local) | 0.83 | 0.18 | 0.19 | 0.30 | 0.18 |
| claude-opus-4-8 (frontier proxy, run 1, full-schema re-score) | 0.11 | 0.01 | 0.04 | 0.04 | 0.05 |
Path-correctness tells a starker story about capability. phi3 never got above 0.04, gemma4:26b never above 0.08, and the frontier proxy, claude-opus-4-8 run through Claude Code subagents, sat between 0.55 and 0.60 on the first run. So my summary of the model side is that a small local model invents a path for most fields no matter what you hand it, a mid-size local model invents far less once it sees even a partial path list (from 0.83 down to between 0.18 and 0.30), which is a stronger result than it looks given how little of the schema that list covered, but still rarely lands the exact one, and a frontier model gets more than half right while still inventing some paths.
Two details in that table matter more than the headline numbers. The first is the control, where the concept note and the wrong-class note score within a point or two of each other at both local tiers and on the first frontier run (0.72 against 0.72 for phi3, 0.19 against 0.18 for gemma4:26b, 0.04 against 0.05 for the frontier proxy), while on the second frontier run the wrong-class note came out well ahead (3.5% invented against 12.1%) and on the third the two tied, which says the prose content of the grounding did no measurable work. The LLM-assisted OCSF mapping essay makes that argument at length, and it points to the schema, more than anything the prompt explained, as what was doing the safety work.
The second is run-to-run variance at the frontier, which came out larger than the single first run suggested. The frontier leg was re-run twice more on the same 705 prompts (141 fields by five conditions), and because the frontier runs went through Claude Code subagents batched one condition at a time, with harness defaults and no temperature or seed control, where the local leg was single-shot per field at temperature 0, the two legs were not run under identical conditions. Path-correctness came out at 57.6%, 67.2% and 68.5%, and only 490 of the 705 prompts (70%) were mapped identically between the first two runs, which had slightly different instructions (_frontier/REPLICATION-2026-06-14.md, Tier B). Runs two and three used the same method and agree within about a point on their totals, but they map 22% of individual prompts differently, and condition by condition they differ by up to 23 points.
The largest of those gaps is excluded from everything below, because run three's batch on the partial-list condition got 89% of paths exactly right against about 63% for the same run's other four conditions, and it got 16 fields right that no other batch in any run got, against between zero and three for every other batch, and its prompt file carried the gold answer on every entry, so that batch can't be shown to be blind (run two can, from its archived gold-stripped inputs). From here on run three means its four other conditions (564 prompts), and a re-run from gold-stripped inputs would settle whether the batch saw the answers or just hugged the shown list more closely.
Real or invented
Most of the frontier's invalid paths were real OCSF
The frontier runs emitted some real OCSF 1.8.0 paths that sit outside the checked-in subset, and the June scorer counted those as silent errors. A post-hoc re-score on 2026-11-16, run by an agent session in my lab with no model called, fetched the full OCSF 1.8.0 class and object definitions from the version-pinned schema API (all 178 objects, profile-inclusive) and re-validated every cached June prediction by walking its path through the real schema (ocsf-deterministic-mapper/results/RESULTS-FRONTIER-CONSTRAINED-2026-11-16.md, Tier B), and a separately written walker got the same numbers from the raw files.
Two of its findings are about the harness, and both apply to anyone building a validator. Twenty-four of the 141 gold mappings (22 distinct paths) are not in the closure the scorer built, among them the CrowdStrike evidences.process.* paths, Okta's src_endpoint.autonomous_system.* and PAN-OS's src_endpoint.owner.groups.name, because the subset's own provenance note says to accept a deeper leaf under an object the subset doesn't hold, and the mapping scorer was written without that rule. So a frontier answer that matched the gold exactly on one of those fields was scored as correct and as a silent error at the same time, which happened 28, 81 and 85 times across the three runs. And two of my gold paths, src_endpoint.hw_info.model and dst_endpoint.hw_info.model for PAN-OS's device-ID fields, don't exist in OCSF 1.8.0, because the hardware-info object has no model attribute, and the same deep-accept rule is what let them into the key, so the answer key carried the exact error it was built to catch.
With the full schema as the validator, the frontier proxy's silent-error rate splits into two very different things (share of prompts per run).
| Frontier run | June scorer "silent" | Path invented (not in full 1.8.0) | Real 1.8.0 path outside the subset |
|---|---|---|---|
| Run 1 (705 prompts) | 17.0% | 5.0% | 12.1% |
| Run 2 (705 prompts) | 26.7% | 9.1% | 17.6% |
| Run 3, four conditions (564 prompts) | 27.7% | 9.9% | 17.7% |
So across the conditions the frontier model truly invented a path on roughly 5 to 10% of prompts (4.5 to 8.5% if I set aside the two paths my own key also got wrong, which the model reproduced on 3, 10 and 8 prompts), and most of what the June scorer called silent error was the scorer's gap. The local models barely move under the same check: phi3 on the partial-list condition goes from 0.69 to 0.68, and gemma4:26b, re-scored from its 705 cached predictions, comes out the same at every condition, so the ladder's shape holds and only the frontier magnitude changes. That is also a partial answer to a tension with the LLM-assisted mapping essay, which reports that GPT-4 and Claude Sonnet 3.5 stayed inside the OCSF vocabulary I gave them across twelve projects: the real invention rate at the frontier is a good deal lower than the June number suggested, though not zero. What the bench can't tell me is whether a complete path list would have closed the rest, because the remaining inventions sit mostly under src_endpoint, dst_endpoint and url, exactly where the partial list showed nothing, so a prompt carrying the full closure is an open leg beside a current model and a second provider.
The alias table
The mapper that cannot invent
The alternative is deliberately unclever. It is a table of generic field-name alias rules, where src_ip and client_ip and SourceAddress all go to src_endpoint.ip, not tuned to the gold, with one hard rule: it may only emit a path that exists in the OCSF 1.8.0 subset, and if no rule fires it marks the field unmapped instead of guessing (ocsf-deterministic-mapper/README.md, Tier B). It is also cruder than I'd ship, because it matches substrings of the whole field name, so ts fires on packets and sends it to time, and CrowdStrike's event. envelope fires the event rule on every CrowdStrike field, and that crudeness shows in the numbers below. Scored on the same 141 fields:
| Mapper | Silent-error | Path-correct | Coverage (fields attempted) |
|---|---|---|---|
| Deterministic alias rules, schema-constrained | 0.00 | 0.16 | 0.67 |
| phi3, best condition (discipline) | 0.60 | 0.04 | 1.00 |
| gemma4:26b, best silent-error condition | 0.18 | 0.06 to 0.08 | 1.00 |
| Frontier proxy, three runs, full-schema re-score | 0.05 to 0.10 | 0.58 to 0.67 | 0.90 to 1.00 |
The zero is true by construction, which is why I trust it and also why it says nothing about whether the paths the mapper does emit are the right ones. A mapper that can only choose from the schema can map a field correctly, map it to a valid but wrong path, or leave it unmapped, and it can never write to a column that does not exist. Its coverage varies a lot by source, from 0.90 on CrowdStrike Detection Summary down to 0.40 on Cisco Umbrella DNS, and its path-correctness across all of a source's fields runs from 0.50 on Cisco ASA down to 0.07 on Okta (results/results.json), which partly reflects how conventional each vendor's field names are and partly the matcher, since CrowdStrike's high coverage and low accuracy come mostly from that event. prefix sending eighteen of its fields to the same event-code path.
The wall
Putting the model behind the same wall
If the constraint is doing the safety work, the obvious next question is whether a model behind the same constraint gets the safety and keeps its advantage. In June I ran that for phi3 two ways, rejecting any invalid path to unmapped, and snapping an invalid path to the nearest valid one (results/results-constrained.json, Tier B).
| Arm | Silent-error | Path-correct | Coverage |
|---|---|---|---|
| phi3 raw (partial path list in prompt) | 0.69 | 0.007 | 1.00 |
| phi3 behind the constraint, reject | 0.00 | 0.007 | 0.31 |
| phi3 behind the constraint, snap | 0.00 | 0.021 | 0.65 |
| Deterministic alias rules | 0.00 | 0.163 | 0.67 |
Behind the wall phi3 is safe, which is what the constraint guarantees for any model, and in the snap arm it gets about an eighth as many paths right as the alias table (0.021 against 0.163) at about the same coverage. It isn't a strict subset of what the rules already get, since two of its three correct paths (Umbrella's query_type and Zscaler's contenttype) are fields the alias table missed, but at this capability tier the constraint is doing nearly all the useful work.
The same re-score put the frontier proxy's cached June answers behind the same reject and snap wall, once with the wall built from the June subset and once from the full 1.8.0 schema, across every condition kept (nothing was pre-registered for this, so all of them are reported rather than one picked), and the results were these.
| Frontier proxy | Run 1 (705 prompts) | Run 2 (705 prompts) | Run 3, four conditions (564 prompts) |
|---|---|---|---|
| Raw path-correct (coverage) | 57.6% (90%) | 67.2% (100%) | 63.3% (100%) |
| Behind a full-1.8.0 wall, reject: path-correct (coverage) | 57.2% (85%) | 65.8% (91%) | 61.9% (90%) |
| Behind a full-1.8.0 wall, snap: path-correct (coverage) | 57.6% (90%) | 67.1% (100%) | 63.3% (100%) |
| Behind the June subset wall, reject: path-correct | 53.6% | 55.7% | 51.8% |
Silent-error is zero in every walled arm, and behind a wall built from the full schema the frontier model keeps nearly all of its accuracy, within a point and a half, against the alias table's 16%, which doesn't move when the alias table is also given the full schema. The whole of that point and a half is the two gold paths that don't exist in OCSF 1.8.0, which the frontier model reproduced and the wall rejected, so the validator caught an error my answer key and the model shared. Apart from that, the accuracy surviving is by construction, since rejecting a path can't take away a right answer when the right answer is a real path, so the measured findings are the coverage cost (from 90 to 100% raw down to 85 to 91% under reject) and the size of the gap to the rules, which makes a frontier model behind a full-schema validator the best of the three designs measured here. It is still one model from one provider on one task, re-scored after the fact, and a current model and a second provider are the runs I'd want before I'd say so on a vendor call.
Two other results from that table are worth more to a practitioner than the headline. Behind the June subset wall, reject cost the frontier model 4 to 12 points of accuracy, because the incomplete schema rejected answers that were right, so a validator is only as good as the schema build it was made from, and a hand-transcribed subset is a liability once a model knows more of OCSF than the subset does. And snap rarely helps: with the full wall it moved 35, 64 and 56 invented paths to the nearest valid one and landed 3, 9 and 8 of them on the gold, which means snapping mostly converts an invented path into a valid wrong one, trading a failure a validator can see for one it can't.
Wrong but valid
The silent error the metric does not count
Here is the part I think matters most for anyone evaluating a product, and it is a limitation of my own metric. Silent-error, as I defined it, counts invented paths, but it does not count a field mapped to a path that exists and is the wrong home, and that failure also validates, also ships and also leaves a detection blind. The deterministic mapper attempted 94 of the 141 fields and got 23 exactly right, so 71 went to a valid path other than the gold, and an agent session in my lab went through all 71 one by one against a three-way rubric: defensible if a reviewer reading OCSF 1.8.0 would accept the path as a home for that field's meaning, lossy or ambiguous if it is the right object or role but drops part of the value or collides with another field, and wrong otherwise. Seven came out defensible, four lossy or ambiguous, and sixty wrong, which is 60 of 141 fields going to a valid path that a validator will pass and a detection will misread (results/alias-wrong-but-valid-adjudicated-2026-11-16.csv, Tier D, single reader). Many of those sixty are the substring matcher's doing, and when the same rule table was re-run matching whole tokens, with camelCase split and CrowdStrike's envelope stripped, 17 of the 60 still landed on the same kind of valid wrong path (10 if camelCase is left whole), so the count describes this crude table more than alias mapping in general (alias_whole_token_check_2026_11_16.py, Tier B). The survivors aren't subtle once you look, with PAN-OS src_host landing on device.hostname, which in a network activity event is the firewall that reported it, PAN-OS rule_uuid landing on the rule's name, and Umbrella's query_type landing on query.hostname.
Wrong but valid · the alias table's 141 fields
Some share of valid-but-non-gold mappings are defensible alternatives that exact match penalizes, and that share can be large even in a key I built myself, since in the shipped-pipeline bench my gold key pinned every Zeek connection to a single activity value where the tool derived it from conn_state, and the disagreement was the key's (What OCSF-native actually delivers has that story). The frontier model's own valid-but-non-gold share runs 25 to 28% of prompts across the three runs, and I haven't adjudicated those, so I can't say how much of it is the model being wrong and how much is my key being narrow. What I can say is that no schema validator will settle it, because a schema can tell you a path exists and cannot tell you it is the right path for this field.
So a schema constraint removes one class of silent error completely and leaves the other untouched. Catching the second class takes something that knows what the field means: a gold key for the sources you actually run, a planted-truth test like the one behind this essay, or a reviewer who reads the mapping. The text-to-SQL pilot in the mapping essay makes the same point one layer down, where a frontier model's queries ran and returned a plausible wrong number at a silent-error rate of 0.49 across nine questions, and I think the two results describe the same shape of risk at two layers of the stack.
Buyer questions
What I'd ask an AI-parser vendor
The framing I'm borrowing here, "the harness, not the model", is a phrase I first heard from a pipeline vendor describing its own OCSF mapper (vendor claim about its own product, Tier C, bias flagged), and I think it holds regardless of whose pipeline it describes. It shifts the useful questions away from which model a product uses and toward what the product does with the model's output, so when a vendor demos an AI-generated parser, these are the things I'd ask:
- Is every emitted attribute path validated against the exact OCSF version the product claims, and what happens to a field that fails validation, whether it is rejected, snapped, constrained at generation or written anyway?
- Is the validator built from the full, version-pinned schema, or from a subset or an older build, given that in my own lab one validator built on a subset rejected right answers and another passed a gold key with two paths that don't exist?
- Which OCSF version is the validator pinned to, and what happens to the generated parsers when OCSF moves (1.9.0 shipped on 2026-08-03 with deprecations)?
- Is there a gold key or planted-truth test for my sources, and will the vendor show its wrong-but-valid rate along with its schema-valid rate?
- Is the generated mapping frozen into deterministic code that I can diff and review, or regenerated by the model at run time, where two of my frontier runs using the same method mapped about a fifth of prompts differently?
The last question is the one I'd weight most heavily. A model that drafts a mapping, a validator built from the real schema that rejects invented paths, and a human or a gold key that reviews the draft before it is frozen into code is a pipeline I would run. A model that maps fields at ingest time on every event is a pipeline whose output can change between Tuesday and Wednesday without a commit, and I don't see how a detection team signs off on that.
Where the model sits · two pipeline shapes
Bottom line: on this task, a validator built from the full version-pinned schema took invented paths to zero at every tier, by construction, and behind it the frontier model kept nearly all of its accuracy, 57 to 66% against a crude alias table's 16%, while the local models had little accuracy to keep. The validator supplied the safety and the model supplied the accuracy, and neither one catches a valid mapping that is wrong, which in a single-reader adjudication was 60 of 141 fields for this crude table and is an unmeasured share of the frontier model's 25 to 28%. What I'd hand a buyer today is the five questions above, plus the reminder that a validator only proves a path exists, so somebody still has to check it is the right one.
Sources
Sources and tiers
- Mapping oracle, model ladder and five conditions:
sdw-lab-benchmarks/ocsf-mapping-oracle(June 2026, Tier B). - Frontier three-run replication:
ocsf-mapping-oracle/_frontier/REPLICATION-2026-06-14.md(Tier B; batched subagents, harness defaults). - Deterministic mapper and phi3 behind the constraint:
sdw-lab-benchmarks/ocsf-deterministic-mapper(June 2026, Tier B). - Frontier behind the constraint, full-schema re-validation, the subset findings and the run-3 exclusion:
ocsf-deterministic-mapper/results/RESULTS-FRONTIER-CONSTRAINED-2026-11-16.md(post-hoc re-score of the June predictions, 2026-11-16, with appended corrections, Tier B). - Wrong-but-valid adjudication and the whole-token check:
ocsf-deterministic-mapper/results/alias-wrong-but-valid-adjudicated-2026-11-16.csv(single reader, Tier D) andresults/alias-whole-token-check-2026-11-16.json(Tier B). - OCSF 1.8.0 class and object definitions: schema.ocsf.io version-pinned API, read 2026-11-16 (Tier A).
- "The harness, not the model": a pipeline vendor describing its own mapper (Tier C, bias flagged).
Related
- LLM-assisted OCSF mapping: what the migration tax actually looks like →
- What OCSF-native actually delivers →
- The ground you're already standing on →
- The field-mapping anti-pattern in security data migrations →
- OCSF version drift: what a mapping pack costs per release →
- The lab: the mapping-oracle and deterministic-mapper benchmarks →