How to run a benchmark that doesn't lie.

I want to open with the case that taught me to put this rule above all the others, because it's the one that nearly got past me. I was running a cross-engine scale ladder, the same simple analytical queries executed by ClickHouse's embedded engine, chDB 4.1.8, and by DuckDB, over byte-identical Parquet, the whole exercise built to ask which engine was faster on security-shaped data. Almost as an afterthought I'd put an answer-equality check at the front of the harness, on the principle that there's no point reporting that one engine is 8% faster than another if the two aren't even computing the same thing. Before timing anything, the harness runs each query on each engine once and compares the results, and I expected that check to be a formality. At 100 million rows it failed: a selective count(*) with an equality filter came back from chDB tens of rows short of the count DuckDB computed over the same files, no exception, no warning, no log line flagging a skipped row group. The query compiled, the engine ran it fast, and it returned a number that was simply too small.

The part worth sitting with is the counterfactual. If I'd trusted the timings, and fast is exactly what a performance benchmark is built to reward, the run would have produced a clean, plausible, publishable table showing chDB holding its own against DuckDB at scale, and the missing rows would have ridden along inside a result that looked completely normal. The faster the wrong engine, the more convincing the bad number, which inverts the usual intuition that a quick answer is a trustworthy one. A timing-only benchmark would have laundered a silent wrong answer into a performance win and published it as one. The only reason it didn't is that the gate I almost didn't bother building compared the two answers before it compared the two clocks. The full case, what the bug was and how I isolated it to one engine's equality-pushdown path on the tail row groups of the file, is its own essay (the query engine returned the wrong answer); what matters for methodology is the order of operations. Correctness is the gate, and timing runs only after the answers agree. Everything else in this piece is downstream of getting that order right.

So rule zero is to verify the answer before you trust the clock, which on generated data is nearly free because you know the ground truth by construction, and on real data means computing a trusted reference answer once and pinning it. Run the same query on every engine and on the reference, fail loudly the moment any two disagree, before a single timing is recorded so a divergence stops the run rather than getting averaged into a result. One refinement the lab handed me later is that "disagree" needs a type-aware definition: the same column of floating-point values summed by five engines lands on three subtly different totals from ordinary IEEE-754 rounding, all of them correct, while the integer counts and sums agree to the last bit, so the gate compares integer-typed answers exactly and floating-point ones within a tolerance rather than crying wolf over benign rounding. The rest of these rules make the speed number honest once you've established that the answers it describes are real.

When you compare two formats or two engines, the thing you most want to hold constant is the data, and the trap is assuming that two tools writing "the same data with the same compression" have actually produced comparable files. They haven't. On identical input at the same codec, PyArrow wrote a Parquet file of 193 MB where DuckDB's writer produced 114 MB, a difference of roughly 1.7× from the encoder alone, with nothing changed about the logical rows or the compression algorithm named on the tin. The writer chooses dictionary encoding thresholds, page sizes, row-group boundaries, and how aggressively it applies run-length and delta encodings, and those choices move the file size and the read cost enough to swamp the format-versus-format difference you set out to measure. If you let each engine write its own files and then time the reads, you aren't comparing the formats, you're comparing the writers, and you probably can't tell which.

Config parity, every encoding knob matched across both writers, is the partial fix and it's worth doing, but matching every knob you can name still leaves the ones you can't, so the cleaner answer is to remove the writer from the comparison entirely. Write the data once, then register those same physical bytes into both catalogs. For an Iceberg-versus-DuckLake read comparison this is concrete: you point Iceberg's add_files and DuckLake's ducklake_add_data_files at the byte-identical Parquet, so both catalogs describe the same files on disk and the only thing left varying is the engine's read path. Done that way, Iceberg and DuckLake came out read-neutral on identical bytes, which is the true result, and one you can only see once you've stopped accidentally benchmarking two encoders against each other and calling it a format comparison.

The general form of the rule is that the variable you're testing has to be the only variable that moves, and "same data" is doing a lot of quiet work in that sentence. Same logical rows is not the same as same bytes, and same codec is not the same as same encoding, so when you can arrange to feed both sides the identical files you should, and when you can't you should at least know which uncontrolled knob is carrying your result.

The machine you measure on has settings that move your numbers, and the one that surprised me most was the operating-system power plan. On this Windows host running the benchmarks under WSL2, switching from the default balanced plan to High Performance dropped the coefficient of variation on a sustained workload from 5.8% to 0.8%, and on a short workload from 19.8% to 2.7%. That is not a small effect, and it came from a setting rather than from hardware, no faster CPU, no more memory, no migration to a proper bare-metal box. The balanced plan was throttling the clock between bursts and ramping it back up unevenly, so each run started from a slightly different frequency state and the timings scattered accordingly. Pinning the plan to High Performance kept the clock steady, and the steadier clock made the measurement repeatable.

The general lesson is that "control the environment" means more than closing your browser, because the modern stack is full of adaptive behaviors that trade steady-state performance for power and that stay invisible until you go looking for the variance they cause. CPU frequency scaling, turbo boost, thermal throttling, the laptop-versus-plugged-in governor, the WSL2 memory and CPU caps, the filesystem the temp files land on; any of these can put a wobble in your numbers that you'll misattribute to the engine. The honest move is to fix the ones you can, document the ones you can't, and report the coefficient of variation so the reader sees how steady the platform was while you measured. A benchmark run on an unpinned laptop power plan is reporting the governor's mood as much as the engine's speed.

I'll flag the obvious caveat here because it's the honest one: a setting that cut my variance by an order of magnitude on this host might do less on a server with a fixed clock, and the specific numbers are this machine's. The transferable part is the instruction to go find the adaptive behaviors before they find you, and to treat the environment as something you configured on purpose rather than whatever the laptop happened to be doing.

Everything above is about the benchmark you ran yourself, where the only thing standing between you and an honest number is your own discipline. There's a second reason a published benchmark can't be taken on faith, and it has nothing to do with method and everything to do with who is permitted to measure at all. Section 1.2(v) of Splunk's General Terms, the agreement every Splunk Enterprise customer is bound by, prohibits using the product to analyze, test, characterize, inspect, or monitor its performance for competitive purposes, and the Splunk Software License Agreement adds Section 3(f), which prohibits providing benchmark results to third parties without prior written consent. Both clauses bind, and Section 1.2(v) is the broader of the two because it restricts the act of running the comparison and not only the act of publishing the result. The practical effect is that a Splunk customer who wants to evaluate whether ClickHouse, StarRocks, or Trino could run their security analytical workload faster, on their own data, on their own hardware, in their own environment, is contractually prohibited from doing it, and prohibited is the operative word here rather than merely discouraged.

This isn't a quirk of Splunk's contract drafting. Oracle's database licensing has carried a benchmarks clause since the original "DeWitt clause" in the 1980s, named after David DeWitt, the database researcher Oracle prevented from publishing comparative benchmark results, so proprietary data infrastructure has been written this way for forty years. The newer open-platform vendors are the exception worth noting, because Snowflake's Acceptable Use Policy does not restrict benchmark testing and Databricks' Master Cloud Services Agreement carries no comparable competitive-testing clause, which means the schema-on-read SIEM you would be migrating away from is far more likely to bind you than the lakehouse you would be migrating toward. When customers can't legally produce comparative benchmarks against their current platform, the only benchmarks that exist are the ones produced by vendors who never agreed to the bound platform's terms. A startup competing with Splunk publishes "we run security workloads X× faster than Splunk" because the startup never signed Splunk's General Terms, while the Splunk customer who could verify or refute the claim with their own data did sign, and is locked out. So almost every "X% faster than Splunk" benchmark in circulation came from a vendor whose product appears favorably in the result, and the class of benchmark that structurally cannot exist under the current contract regime is the customer-driven head-to-head on real workloads, published openly.

The clearest worked example of how that incentive bends a number comes from outside security, from the streaming-infrastructure market, where the contract regime is looser but the vendor-benchmark dynamic is identical, so the mechanics show through cleanly. Stanislav Kozlovski, an Apache Kafka committer and ex-Confluent engineer, published a forensic read-through in November 2024 of WarpStream's vendor cost calculator, and the calculator, in the version he analyzed, made Kafka look ~3× more expensive than what an experienced operator would actually pay on AWS for the same workload (1 GiB/s sustained, 7-day retention, replication factor 3). The drift came from five mechanical sources, each individually defensible and collectively directional: the calculator selected r4.4xlarge instances when r4.xlarge would have sufficed at the modeled load; it assumed gp2 storage at $0.10/GiB when sc1 HDD at $0.015/GiB was appropriate for the warm-tier workload; it silently shifted the compression-ratio assumption from 5:1 to 4:1 in a quiet update, inflating Kafka costs ~25% with no change to the WarpStream side; it provisioned 60% free space on each broker disk where 50% had been the prior default; and the visible JavaScript multiplied replication cost by an extra (RF-1) factor, double-billing replication traffic at RF=3. Kozlovski's fair AWS Kafka cost for the same workload, before any optimization, came to ~$337k/month against the calculator's $1.264M, a 3.75× delta, and with KIP-405 Tiered Storage and KIP-392 Fetch From Follower the optimized Kafka cost dropped to ~$167k/month, so the headline "WarpStream is 10× cheaper than Kafka" survived in AWS only at 1.3× and reversed sign in GCP (Kafka 21% cheaper) and Azure (Kafka 77% cheaper) at the same workload. Whether WarpStream is a good product is beside the point, because the issue is that the calculator was produced by the vendor who benefits from the comparison, by a methodology the vendor controls, with no independent reviewer who could have pushed back on any of the five drifts before it shipped. None of them required malice, only the ordinary gravity of incentive, and that is exactly the failure mode Section 1.2(v) protects in security, where the customer who could verify can't, the vendor who could be wrong faces no structural check, and the published number becomes the industry reference anyway.

What's instructive is Kozlovski's response, because rather than publish a counter-narrative he shipped a tool, the AKalculator, with the stated principle that it's "incentivized to show the lowest possible cost for Apache Kafka that's also realistic" and the methodology fully disclosed, every assumption visible, every default toggleable, every cost line item attributed to a specific price-sheet entry. The AKalculator's bias is the opposite of the vendor calculator's and he says so on the page, which doesn't make the tool neutral, because nothing is, but it lets the reader adjust, which is the thing a closed methodology cannot offer. Translate the same dynamic to security and the structural problem becomes plain: when the "ClickHouse is X× cheaper than Splunk" benchmark drops, every one of the five Kafka-calculator drift modes is available to the publisher (instance sizing, storage class, compression assumption, free-space allocation, replication accounting) plus the security-specific ones (ingest-source mix, query-suite composition, retention-schema-tax accounting, OCSF normalization cost), the Splunk customer who could run the comparison is contractually prohibited from doing it, and the practitioner who could publish a counter-tool with disclosed methodology faces the same Section 1.2(v) restriction. The fix isn't another counter-benchmark from a differently-incentivized vendor, because what's needed is an independent measurement layer that doesn't share either side's incentive in the first place.

The instinct to build that layer is "open-source the benchmark," push the methodology and the Docker Compose definitions and the data generators to GitHub and let the community verify, and that instinct is right about the methodology, which should be open, but it's wrong about the reference implementation, because the executable artifact, in a comparison set that includes commercial software with restrictive licensing, can't ship publicly without putting either the publisher or the downloader in contract violation. What does work is publishing the methodology, hardware spec, query suite, and result openly under a practitioner brand, while the reference implementation is shared under a one-page mutual NDA with engagement prospects and qualifying reviewers, and the whole thing goes through an annual external review by a named practitioner with relevant standing (a security data engineer, an OCSF contributor, an analyst-firm researcher) under NDA, with their signoff published on the public methodology page. That structure preserves the audit trail that makes a result credible without forcing the publisher to violate the licensing terms of the platforms they're characterizing, and it's close to how the TPC suite and MLPerf already work: both publish methodology openly, neither distributes a turn-key reference implementation a random customer can clone and run against arbitrary commercial software, and both rely on qualified-implementer audit cycles and independent third-party review. Security data has been waiting for the equivalent layer for the last decade and hasn't gotten it, and the contract structure that suppresses it is part of why.

The fourth requirement the TPC analogy understates is the one my own lab embarrassed me into, which is that the benchmark has to verify the answer and not just the clock, because open methodology is no protection against an unverified result and a performance test without a correctness gate can certify the wrong number as a win the moment the wrong engine happens to be the fast one, which is rule zero arriving from the other direction. So a practitioner weighing a benchmark someone hands them has a short checklist: read your own platform's actual contract first, because Splunk's clause is about ten minutes to find at splunk.com/legal and knowing what you can and can't do under your existing license is the precondition for any comparison conversation; discount vendor-published comparative benchmarks accordingly, because "X is 10× faster than your SIEM" reads as a marketing claim rather than a measurement when the customer can't independently verify it; and when an independent benchmark is offered, ask whether the methodology actually publishes openly, whether the result is reproducible by someone running the reference implementation under appropriate licensing, and whether a named external reviewer has audited the methodology under NDA and signed off publicly. Those questions are what separate independent measurement from vendor marketing dressed up as independent measurement, and most published security-tool benchmark coverage I've looked at fails at least one of them.

Sources, all Tier B (Apache Kafka committer + ex-Confluent engineer + tool published with fully disclosed methodology): Stanislav Kozlovski, "The Brutal Truth about Apache Kafka Cost Calculators" (long-form analysis); "no one will tell you the real cost of Kafka" (cross-AZ networking deep dive); AKalculator (the counter-tool with disclosed methodology); KIP-405 (Tiered Storage) and KIP-392 (Fetch From Follower) as the operator-controllable cost-reduction primitives the analysis quantifies. Contract terms quoted verbatim from Splunk General Terms Section 1.2(v) and SLA Section 3(f); Oracle DeWitt-clause history, Snowflake AUP, and Databricks MCSA are public.

I should be precise about what these results are, because the discipline I'm arguing for applies to me first. This is single-machine, first-party, Tier B work. The absolute milliseconds are this host's and nobody else's, the 1.7× PyArrow-versus-DuckDB file-size gap is from these particular writer defaults at these versions, and the power-plan variance numbers are a property of this Windows and WSL2 setup. I'd push back on anyone, including me, who quoted the raw numbers as a server-class result or a universal constant. What transfers is not the milliseconds but the ratios and, more durably, the method: the coefficient of variation has a scale below which your comparison is noise, encoders make "same codec" an incomplete control, contention inflates variance, adaptive power behaviors wobble an unpinned clock, and parallel writers make byte hashes the wrong integrity check for Parquet. Those are properties of the measurement problem, and they hold whatever host you run on.

The closest analogues for what a credible benchmark looks like come from outside security, in the TPC suite and in MLPerf, both of which publish their methodology openly and subject results to audited review rather than asking you to take a vendor's table on faith. Security data has lacked the equivalent, an independent layer with open methodology and a named reviewer, and as the contract section above lays out the reasons it's lacked one are partly contractual, because the schema-on-read SIEMs prohibit customers from running competitive performance tests at all, so the only benchmarks that exist are vendor-funded by construction. The method rules are the other half of that argument, because even when no contract is stopping you, even on open engines you ran yourself, the benchmark will still mislead you if it skips rule zero, since open methodology you can read is no protection against an answer nobody verified.

So the method is the deliverable, more than any single milliseconds figure, and that's deliberate. A speed number ages the moment the next engine version ships, but a harness that gates on correctness, reports its noise, controls its data and its environment, and hashes the right thing keeps producing numbers you can stand behind across versions and hosts. The numbers are evidence; the method is what makes them admissible.

Evidence: Tier B (first-party, single machine, reproduced). Findings drawn from SDW Lab runs on a Windows/WSL2 host: the cross-engine answer-equality divergence (chDB 4.1.8 vs DuckDB over byte-identical Parquet); coefficient of variation up to 55% at 1M rows settling to ~5% at 10M and ~4% at 100M; PyArrow 193 MB vs DuckDB 114 MB at the same codec on identical data; the High-Performance power plan dropping CV from 5.8% to 0.8% (sustained) and 19.8% to 2.7% (short); and Parquet's lack of default byte-reproducibility from parallel row order, with the reproducible single-threaded / ORDER BY layout running roughly 20% smaller. Absolute timings are this host's; the ratios and the method transfer. Methodology is published with the lab.