The method · Research
Hunt for contradictions before confirmation.
Most architecture evaluations form a hypothesis, find evidence that supports it, and declare victory, so this practice flips that order, because the supporting evidence is the easy part while the contradictions are where the position actually gets stress-tested. In the first 25 hypotheses I ran this way, in 2025, that flip changed roughly 60% of the central claims.
The flip
A hypothesis is only worth something if you tried to break it.
The default research pattern in this industry runs in one direction, where you form a position ("ClickHouse will give us sub-second security analytics") and you go gather supporting evidence from vendor benchmarks, case studies, and analyst reports, so within a week you have a stack of corroboration and a recommendation that feels rigorous even though it isn't.
Confirmation bias makes the supporting evidence cheap and the contradictory evidence expensive. The contradictory evidence sits in places the marketing surface won't show you. GitHub issues where practitioners describe what actually broke in production, academic papers that characterize the failure modes, framework developers who acknowledge the limitations their vendors don't mention. Finding it requires looking specifically for it.
The flip this practice runs:
- Form the hypothesis. Specific, falsifiable, with a number where possible. "ClickHouse runs security workloads 100× faster than the dominant schema-on-read SIEM" is testable; "ClickHouse is good for security analytics" is not.
- Gather supporting evidence. The easy step. Time-box it.
- Hunt for contradictions. The hard step. Search GitHub issues for the technology in question. Read the academic literature on its failure modes. Find independent analysts whose work isn't vendor-funded. Engage framework developers, since the people who built the thing have less reason to oversell it than the people selling it.
- Engage practitioners and developers directly. Production deployments are the only environment where vendor claims meet real workloads. The people running them are the highest-quality source.
- Synthesize a balanced assessment. Both perspectives in the same place, with the central claim and the conditions under which it fails.
The process started on a 2025 sample of 25 hypotheses, and it produced a public, versioned literature review that now holds 273 catalogued sources, each carrying an evidence tier and a verification verdict. In that first 2025 sample I recorded contradictions for roughly 80% of the vendor technology claims, and around 60% of the vendor performance claims needed significant contextual qualification or outright revision while the other 40% held up. Those shares are a historical tally from an early synthesis rather than a current measurement, and I'd read them as direction only, but the lesson they pointed at still holds, which is that vendor claims aren't uniformly wrong, so the central claims have to be tested individually, because many of the testable ones move once you actually test them.
Five patterns
What the first 25 hypotheses (2025) revealed about how vendor claims work
1. Performance claims are selectively true.
The ClickHouse "sub-second on billion-row datasets" claim is technically correct. Cloudflare's production deployment runs ClickHouse at 6 million requests per second. My own lab measurement against a schema-on-read SIEM found two regimes rather than one headline: the index wins the simple lookups, and the lakehouse engines win the hunting-shaped aggregations by 5–62×, with 46.8× on native ClickHouse and 10.1× over open-format Iceberg (Tier B, single host, identical answers), which is a large advantage but a long way short of the 100–1000× figures that circulate. The question I'd ask next is what the access controls a multi-tenant SOC needs do to those numbers. The lab's row-level-security bench measured only the engine-side cost, on DuckDB 1.5.3, and found it small and mixed (from +81% on a sub-millisecond count to −86% on a top-talkers query that simply reads fewer rows), while the catalog-layer enforcement that Polaris, Unity Catalog or Lake Formation adds is outside that bench, so that is where the penalty, if there is one, still has to be found.
The pattern is that vendor benchmarks highlight peak performance under benchmark-friendly conditions, while real-world value comes out of sustained average performance across mixed workloads, with the security controls and operational requirements real environments carry, so both numbers can be defensible and the question is which one is being put to work.
2. Implementation timelines are universally underestimated.
In the 2025 sample, vendor timeline estimates ran 2–5× short of what practitioners reported in every technology category I looked at, and the "AI agent for defensive security" category was the worst. The table below is that sample's working tally, not an independently sourced measurement, so read it as Tier D direction rather than as numbers to plan against.
| Category | Vendor-quoted | Practitioner-actual |
|---|---|---|
| Security data pipeline platforms | 2–3 months | 6–8 |
| CAASM (cyber asset attack surface management) | 3–4 months | 8–12 |
| EPSS (exploit prediction scoring system) integrations | 1–2 months | 6–12 |
| "AI agent for defensive security" | "immediate" | 6–9 months once human-in-the-loop oversight requirements are accounted for |
The mechanism is structural, because vendors optimize their demos for time-to-first-value with sanitized data, pre-configured environments, and zero integration complexity, while real deployments carry legacy integrations (absent from the demo), custom security requirements (absent from the benchmark), organizational change management (absent from the SOW), and skills development (absent from the TCO model), so a TCO that doesn't include the things that take the most calendar time arrives at the wrong calendar.
3. Some claims hold up under scrutiny.
The reduction in downstream SIEM licensing costs from a security data pipeline platform held up better than most claims in the sample. The research page carries it as H1-COST-02, a 50–80% cut to the licensing line at Tier C and 3 of 5, anchored to the pipeline vendors' own published customer reductions and my MOAR cost model, and the market signal is consistent with it, since Cribl raised its Series E at a $3.5B valuation in August 2024 (Cribl's own press release). It is the licensing line item and not the all-in cost, because the pipeline platform and the routing work carry their own cost, but the direction of the claim is defensible.
What distinguishes the durable claims from the inflated ones is independent corroboration across multiple production deployments, transparent acknowledgment of implementation challenges, and TCO models that include migration costs, training, and ongoing operational requirements, so mature vendors and durable categories let those numbers travel while the inflated ones can't.
4. Framework developers offer the most balanced perspective.
The best validation across the portfolio came from engaging framework creators directly, the people with deep technical expertise and no sales quota. Jay Jacobs at the FIRST EPSS SIG put the EPSS effort-reduction claim at 70–80% (slightly under the 85% marketing number), with the caveats that EPSS can't predict zero-days, requires 6–12 months of organizational calibration, and needs professional services for threshold tuning in the large majority of organizations, so the effort reduction is real but it isn't "plug and play."
Peter Kaloroumakis at MITRE D3FEND clarified that "Wall Theory" (a derived interpretation showing up in some practitioner literature) isn't actually MITRE-endorsed methodology, even if the 45–75% defensive improvement number is achievable through custom implementation with 3–5 dedicated staff over 18–30 months. The framework developer acknowledged what was possible and what was being asserted as official; the marketing literature blurred the two. The D3FEND Wall on this site is a different thing from the "Wall Theory" he was asked about. It is my own visualization, labelled as an original framework, built directly from D3FEND's published ontology to show which defenses share an artifact with which attacks, and it makes no defensive-improvement claim and carries no MITRE endorsement.
The pattern is that framework developers acknowledge both capabilities and limitations while vendors emphasize capabilities and downplay limitations, so the two perspectives lead to different architecture decisions, which is why the framework developer tends to be the better source on what the framework can do.
5. AI claims need the most skepticism.
The hypothesis that started the AI agent thread of the research: "90% success rate in administrative security tasks, 60% talent reduction." The contradictions surfaced quickly, because large language models generate plausible-but-incorrect security advice routinely (the hallucination risk) and advanced models exhibit deceptive behavior under adversarial pressure (the goal-misalignment risk), so human oversight on security-critical decisions is structural rather than a nice-to-have.
The revised position I took in 2025 was 70–85% success rates with mandatory human-in-the-loop oversight, and 40–50% talent reduction once the oversight workload is included. Those two ranges were working estimates rather than measurements, and I'd now hold them at Tier D, since most published success-rate figures in this category are vendor-sponsored and the first-party evidence I trust more, such as the lab's measurement of how often a model's wrong answer arrives quietly, measures something narrower. What survived is the posture, with audit trails, approval workflows, and verification systems treated as non-negotiable, because the AI agent capability is real while the "fire-and-forget autonomous SOC analyst" framing is dangerous, so the right posture treats AI as augmentation that requires oversight rather than as replacement.
The category-level pattern is that AI marketing is moving faster than AI capability, so the gap between promise and production is wider here than anywhere else in the portfolio, and the contradictions surface fastest in adversarial-research literature rather than in vendor case studies.
What this means for the people buying this work
Four practices that move the floor on architecture decisions
Stop accepting vendor benchmarks unverified.
Every vendor benchmark is a central claim, so test it against GitHub issues for the technology you're evaluating, independent analyst work that wasn't vendor-funded, practitioners who hit production at scale, and framework developers for the canonical reading on what the framework can and can't do, and the specific question to ask is what's the workload shape under which this number stops being true, because if the vendor can't answer it, the number isn't doing the work the proposal needs it to do.
Plan for 2–3× vendor timelines.
When the vendor says 3 months, budget 6–9, and when the vendor says no consulting required, budget for outside expertise, and when the vendor says "plug and play," expect significant custom development, because the pattern held across every category in the 2025 sample, so treating it as the default, while the numbers behind it stay a working tally rather than a measurement, saves months of replanning later.
Demand balanced evidence in the procurement conversation.
The four questions that separate transparent vendors from ones to walk away from:
- What are the top three reasons customers fail to achieve these results?
- Can I speak with a customer who experienced implementation challenges?
- What are the documented limitations of this technology in production?
- What organizational maturity prerequisites must be met?
Vendors who can answer these are the ones whose numbers are worth more than the marketing they're attached to. Vendors who deflect are telling you something about the gap between the claim and the reality.
Build contradiction discovery into the standard process.
Make it a structured part of every architecture evaluation rather than a stretch goal, so that week one gathers supporting evidence (the traditional approach), week two hunts contradictions deliberately (GitHub issues, academic literature, independent analysts), week three synthesizes both perspectives into a balanced position, and week four validates with experts like framework developers and production practitioners, which produces evidence-based decisions that survive contact with deployment instead of confirmation-biased ones that show their problems once contracts are signed.
The method is what makes the research portable.
The twelve anchor hypotheses, the contradictions log, and the running update history are all on the research page. The thesis page connects them to the program POV.
