The better the model, the quieter the wrong answer.

I built one battery of questions against a single OCSF event store and answered each question six ways: plain LLM text-to-SQL over the raw tables, two flavors of GraphRAG (retrieve context, let the model compose), an LLM writing SPARQL against a curated ontology and executing it deterministically through an Ontop OWL2QL rewrite, a deterministic OBDA template, and a hand-curated metrics layer of the kind a mature data team ships. The LLM arms ran at three model tiers, Haiku, Sonnet, and Opus, with the model held constant across arms within a tier, so I'm comparing methods rather than mixing a strong model into one arm and a weak one into another. Every arm is scored by one shared function that sorts each answer into three buckets: correct, silent (a confident answer that is wrong, with no error raised), and loud (a visible failure such as an empty result, a parse error, or a refusal). Eight trials per cell, because these models are sampled rather than deterministic and I wanted the variance.

The questions split into two kinds, and that split is what the rest of this turns on. Some are lookups: find the encoded PowerShell command on this host, return the no-MFA privilege-escalation event. Those are find-the-needle questions, and text-to-SQL with full table access nails them, correct on every trial at every tier. The other kind is compute-over-population: count the distinct physical assets behind a human's six identifiers once you collapse the hostname and IP and instance-id aliases, compute the dwell time between the first and last event of the chain, reconstruct the stages of the attack in the order they happened. On those three questions, none of the LLM-authored arms produced the correct answer at any of the three tiers; the only correct compute-over-population answer in the whole run came from the hand-curated metric.

So at the level of did it get it right, the comparison is dull: on the questions that require reasoning over the whole population, the model is wrong. What isn't dull is how it's wrong, because that is what decides whether you find out.

If you've shipped anything LLM-shaped you already know the two reflexes for making a model's output trustworthy: let it check its own work by running it, and sample it a few times and take the majority. I tested both as pre-registered arms, with the predictions written down before the runs, and both fail on the compute tail in instructive ways.

The first is the agentic loop. Give the model the same question and schema as the one-shot text-to-SQL arm plus a read-only handle to run DuckDB against the store, and let it iterate up to five rounds: write a query, run it, look at the rows or the error, revise, stop when it's satisfied. The pre-registered guess was that execution feedback would clean up the loud failures, the model fixing its own empty results and syntax errors, without touching the silent ones, because a wrong aggregate runs fine and returns plausible rows, so the loop has no error to correct against. That is what happened, and it's worse than a wash. Loud failures fell to roughly zero. Silent failures went up: the tail silent rate rose from about 0.54 to 0.96 at Haiku, 0.63 to 0.88 at Sonnet, and 0.50 to 0.88 at Opus. The loop's stopping condition is that the query runs and returns something plausible, and on the compute tail that condition is satisfied by a confidently wrong answer, so the loop spends its rounds converting visible failures into invisible ones. I checked one end to end: Opus, after five rounds of building an elaborate identity-closure query, returned one distinct asset against a ground truth of nine, a query that executed, looked confident, and was wrong.

The second is self-consistency. I didn't need new generation for this, since I already had eight trials per question, so I clustered each arm's eight answers, took the majority answer when one cleared five of eight, and abstained otherwise, counting an abstention as a caught failure. The pre-registered prediction was that this catches silent errors only where the model is inconsistently wrong, and the GraphRAG tail is the opposite: it's silently wrong on every trial, in agreement, so the eight wrong answers clear the majority and the ensemble emits a confident silent error. That held. At the frontier, Sonnet and Opus, self-consistency caught none of the three GraphRAG compute-tail questions; all three came back silently wrong with the votes agreeing. And the catch rate decreased with capability, because the better model is more consistently wrong, so its own samples agree more and the ensemble is least useful exactly where it's most dangerous. Where self-consistency did catch things, on text-to-SQL, it was abstaining because the trials disagreed, different SQL each round and different wrong answers, none reaching a majority. That catch is real, but it's a function of disagreement rather than of knowing the answer is wrong, and it cuts both ways: it abstained on the one configuration where Opus occasionally got the tail right, throwing the correct answer out with the rest. Self-consistency produced zero correct compute-tail answers in the entire run.

Both reflexes optimize for the wrong target. The agentic loop optimizes for executable; self-consistency optimizes for agreed-upon. On compute-over-population questions a wrong answer is usually both, so both methods preserve the silent error and one of them amplifies it.

The honest scope keeps this from turning into never let a model near your data. For lookups, find the event, pull the command line, return the record that matches, LLM text-to-SQL was correct on every trial at every tier, and that's most of what an analyst actually asks. The failure is specific to compute-over-population: counts, distinct counts after alias resolution, durations, orderings, the questions where the answer is a number derived from the whole set rather than a row you can point at. Those are also, not coincidentally, the questions detection and incident response lean on hardest, and they are, I'd argue, the ones where a confident wrong number is most costly, because it looks like an answer and shows up on a dashboard with no flag attached.

So the rule I'd write down is to verify every compute path, and to treat an LLM composed this query as a reason for an external check rather than a substitute for one. For the correctness-critical tail that means a deterministic rewrite that refuses what it can't express, a curated metric a human authored and owns, or a typed query layer that rejects the wrong shape, something outside the probabilistic path that can say no. It doesn't mean a bigger model, and it doesn't mean wrapping the model in a self-correction loop or an ensemble of its own samples, because I measured both and they move the failure in the wrong direction. If you can only afford to harden one part of an LLM analytics stack, harden the part that executes and checks, not the part that generates.