Security-specialized AI became a shipping product category in the last two weeks of July. Sakana AI released Fugu-Cyber on 21 July. Google announced Gemini 3.5 Flash Cyber the same day. Microsoft followed on 27 July with MAI-Cyber-1-Flash, its first in-house cyber model. All three are gated rather than generally available, all three are priced or licensed separately from their vendors’ general models, and all three arrived with benchmark numbers attached. Here is what each one actually is, and how to read the numbers, because they are less comparable than they look.
What shipped, and what it costs
Three security-specialized AI launches, and three quite different products underneath similar marketing.
Sakana AI Fugu-Cyber is the third endpoint in Sakana’s Fugu family, after Fugu and Fugu Ultra. It is priced at $6 per million input tokens and $36 per million output, with cached input at $0.60, and it is available only on Sakana’s Token Plan rather than its subscription tiers. Sakana reports 86.9% on the CyberGym benchmark and 72.1% on CTI-REALM.
Google Gemini 3.5 Flash Cyber is described in Google’s own announcement as built on 3.5 Flash and "fine-tuned for finding and fixing cybersecurity vulnerabilities." Access is limited to governments and trusted partners. Notably, Google published no benchmark figure, saying only that the model reaches "competitive performance at the frontier" on CyberGym, and attributing that result to CodeMender, an ensemble that runs multiple 3.5 Flash Cyber agents together rather than the model alone.
Microsoft MAI-Cyber-1-Flash is Microsoft’s first cyber-specific model, built from its MAI-Thinking-1 lineage and running inside MDASH, the multi-model agentic scanning harness Microsoft contributed to the Open Secure AI Alliance last week. Microsoft reports the combined system at 96% on CyberGym, with a claimed 50% cost saving against its previous best configuration. Public preview opens 3 August.
They join OpenAI’s GPT-5.5-Cyber, in limited preview since May and gated to critical-infrastructure defenders, which we covered when OpenAI opened it to trusted cyber partners.
Security-specialized AI: a model, or a system?
Two of these are models. One is not, and the difference changes what a benchmark score tells you.
Google’s and Microsoft’s releases are fine-tuned models: a base model further trained on security work. Sakana’s is something else. Sakana describes Fugu as "a multi-agent system that behaves like a single model," and more precisely as "itself a language model trained to call various LLMs in an agent pool." You send a request to one endpoint and Fugu either answers directly or assembles a team of other vendors’ models to handle it. Which models it picks, and how it coordinates them, are "proprietary, so this routing information is not exposed by design."
That is a legitimate architecture, and our earlier look at the Fugu family covers why it works well for general use. For security procurement it has a specific consequence: the capability you are buying belongs to the whole system, not to a security-tuned model, and the system’s composition can change without notice. A score attached to it is a snapshot of an assembly you cannot inspect.
Microsoft’s MDASH has the same property, orchestrating what Microsoft describes as more than 100 specialised agents. Comparing MDASH to Fugu-Cyber is fair. Comparing either to a single fine-tuned model is not, and the published numbers invite exactly that comparison.
Why the benchmark numbers do not line up
Every security-specialized AI launch this month cites CyberGym, so it is worth knowing what it measures. It is an academic benchmark from UC Berkeley, accepted at ICLR 2026, containing 1,507 real vulnerabilities drawn from 188 open-source projects in Google’s OSS-Fuzz corpus. The agent receives a vulnerability description and the unpatched code, and must produce a proof of concept that triggers on the vulnerable build and not on the patched one. Scoring is mechanical and binary.
The complication is that CyberGym has four difficulty levels, defined by how much information the agent gets. Level 0 gives no description, and the benchmark’s authors measured 3.5% success there. Level 1, the default, adds the description and source. Level 2 adds a stack trace. Level 3 hands over the actual patch.
A CyberGym percentage without a level attached is not really a measurement. Of the figures published across this category, exactly one names its level: Microsoft’s earlier 88.45% result from 12 May, which states it used "CyberGym’s default configuration (level 1)" across the full 1,507 tasks. The later and larger Microsoft numbers do not, and neither does Sakana’s.
There is a sharper version of this problem, and it sits inside a single vendor. Microsoft’s June follow-up restates the Build figure as "96.5% (any crash)". CyberGym counts a task as solved when the generated proof of concept crashes the unpatched build and not the patched one, and the benchmark’s public leaderboard reports its column as reproducing the target vulnerability. "Any crash" is a looser criterion, and it was not attached to the 88.45% or to the Build headline. Two numbers, same vendor, same benchmark, different scoring rules.
Trial count matters just as much and nobody discloses it. Berkeley’s own follow-up data showed one model going from 28.9% on a single attempt to 66.7% across 30 attempts. Same model, same benchmark, more than double the score.
For a sense of how far the vendor figures sit from measured ground, the CyberGym authors’ own public leaderboard, hosted by Berkeley’s Center for Responsible Decentralized Intelligence, currently tops out around 18% for the best model-and-scaffold pairing it lists. None of the vendor numbers appear on it, and the same group states plainly that no comprehensive benchmark currently exists for assessing AI cybersecurity capability. That is not evidence the vendors are wrong. It is evidence that nobody has run their configurations independently, which is a different and more useful thing to know.
The timeline worth noticing
Sakana’s release page carries a headline calling Fugu-Cyber "state-of-the-art," while the body text says its 86.9% is "comparable to leading cybersecurity-focused frontier models such as GPT-5.5-Cyber and Mythos-Preview." Those are different claims, and the second one is the accurate one.
It also lands seven weeks after Microsoft reported 96.55% on the same benchmark at Build 2026 on 2 June, roughly ten points higher. Whether that comparison is fair is impossible to establish, because the two figures disclose different things about how they were produced.
Sakana published no competitor comparison of its own. The 86.9-against-85.6 side-by-side that circulated was assembled by trade press from separate announcements, and the 85.6 is OpenAI’s own figure for GPT-5.5-Cyber in what OpenAI explicitly calls single-model evaluations. That is not the same kind of object as a multi-agent system’s score, so doing arithmetic on the gap implies a head-to-head that neither vendor ran.
That is not an accusation against any vendor. It is the ordinary condition of a security-specialized AI market that is ten days old and marketing on numbers with no agreed method behind them.
Why the disclosure gap is partly structural
Some of this is genuinely hard rather than evasive, and the distinction matters if you are deciding how much to hold it against a vendor.
Offensive security tasks are open-ended in a way most benchmarks are not. There is rarely one correct answer, partial progress is difficult to score, and an agent given more attempts, more tool access or a longer time budget will do substantially better. Two labs can run the same benchmark honestly and produce very different numbers purely from harness choices.
The dual-use problem compounds it. Detailed methodology for an offensive security evaluation is itself sensitive information, so vendors have a real reason to publish a score without the recipe. That reason is defensible. It also makes independent verification impossible, and both things are true at once. We ran into the same asymmetry from the defender’s side when safety guardrails blocked forensic analysis during a live incident response.
Where the independent measurement is happening
The national AI security institutes are evaluating cyber capability, and they publish their methods. The UK AI Security Institute’s assessment of Claude Mythos Preview reported 73% on expert capture-the-flag challenges, noted it was the first model to complete a 32-step cyber range end to end, and then reported that it managed this on 3 of 10 attempts, averaging 22 of 32 steps, and failed an operational-technology range outright. Attempts, averages and failures, all disclosed. In July the UK institute and the US Center for AI Standards and Innovation published a joint assessment of Kimi K3 in the same style.
None of that work uses CyberGym. The benchmark the commercial category markets against currently has no independent-evaluator adoption.
The capability is real even if the scores are not
None of this means the category is vapour, and the strongest evidence for it does not come from any benchmark.
Models are demonstrably finding real vulnerabilities. During the OpenAI evaluation whose models reached Hugging Face’s production systems, they chained stolen credentials and genuine zero-days into a working remote code execution path. In mid-July a Searchlight Cyber researcher pointed a frontier model at a clean copy of WordPress core and it produced the full wp2shell exploit chain in a little over ten hours. Those are outcomes rather than scores, and they are not in dispute.
What has not been established is the narrower claim the benchmarks are being used to sell: that a security-specialized product outperforms a general frontier model wrapped in a good harness. That is the whole commercial proposition of this category, and it is precisely the thing the published numbers cannot currently support.
What to ask before you buy
Six questions separate a measurement from a marketing figure, and none of them is hostile. Put them to any security-specialized AI vendor that shows you a percentage.
Ask which difficulty level a CyberGym score was run at. Ask how many attempts each task was given, and whether the number is a single pass or a best-of-N. Ask whether the full 1,507-task set was used, or a subset. And ask what agent scaffold wrapped the model, and whether it was built for this benchmark specifically.
Then two that nobody markets on. Ask what the false-positive rate is, because almost every published security-AI result measures detection or exploitation success and almost none reports the noise, and alert fatigue is what actually kills security tooling in practice. And ask who on your side reviews the output, because a tool that raises the volume of findings needing expert triage without raising your capacity to triage them makes your position worse rather than better.
Then ask what an independent party measured, because for most of these products the answer is currently nothing.
Something real is being built here. Defenders have had a genuinely hard year, the Hugging Face intrusion showed what happens when safety guardrails block the people doing forensics, and purpose-built defensive tooling is a reasonable response. The products are worth evaluating. The scores are not yet worth deciding on, and the gap between those two statements is where a procurement mistake lives. If you are building the evaluation case internally, the zero-trust posture you already run is a better predictor of outcomes than any vendor’s benchmark table.
Frequently Asked Questions
What is a security-specialized AI model?
It is a model or system built specifically for cybersecurity work such as finding vulnerabilities, generating detection rules or reproducing exploits, rather than a general model applied to security tasks. As of late July 2026 the category includes OpenAI’s GPT-5.5-Cyber, Google’s Gemini 3.5 Flash Cyber and Microsoft’s MAI-Cyber-1-Flash as fine-tuned models, plus orchestration systems such as Microsoft’s MDASH and Sakana’s Fugu-Cyber.
How much does Sakana Fugu-Cyber cost?
Sakana lists Fugu-Cyber at $6 per million input tokens and $36 per million output tokens, with cached input at $0.60 per million. Long-context requests above 272K tokens are priced higher. It is available only through Sakana’s Token Plan, not the $20, $100 or $200 subscription tiers that cover Fugu and Fugu Ultra.
Is Fugu-Cyber a fine-tuned security model?
Not in the way the other launches are. Sakana describes Fugu as a multi-agent system that behaves like a single model, and specifically as a language model trained to call other vendors’ models in an agent pool. It can answer directly or assemble a team. Its benchmark score therefore describes the whole system rather than a security-tuned model, and Sakana states that the routing and pool composition are not exposed by design.
What is CyberGym?
CyberGym is an academic benchmark from UC Berkeley, accepted at ICLR 2026, built from 1,507 real vulnerabilities across 188 open-source projects in the OSS-Fuzz corpus. An agent is given a vulnerability description and unpatched code and must generate a proof of concept that triggers on the vulnerable version but not the patched one. It has four difficulty levels depending on how much information the agent receives.
Why do CyberGym scores vary so much between vendors?
Four undisclosed variables each move the number substantially: which difficulty level was used, how many attempts each task was given, whether the full task set or a subset was run, and what agent scaffolding wrapped the model. Berkeley’s own data shows one model going from 28.9% on a single attempt to 66.7% across 30 attempts. Almost no vendor discloses any of the four, so two scores are rarely measuring the same thing. Scoring rules drift too: Microsoft’s June follow-up qualifies its headline figure as “any crash,” a looser criterion than reproducing the target vulnerability, and that qualifier was absent from its earlier result.
Is AI actually finding real vulnerabilities, or is this all benchmarks?
It is finding real ones, and that evidence is stronger than any published score. During an OpenAI evaluation in July, models chained stolen credentials and zero-days into a remote code execution path on Hugging Face’s production infrastructure. Separately, a Searchlight Cyber researcher directed a frontier model at WordPress core and it produced the full wp2shell exploit chain in around ten hours. The open question is not whether AI can find vulnerabilities. It is whether a specialized security product does it better than a general frontier model with a well-built harness, which is exactly what the benchmarks are being used to claim and exactly what they cannot currently show.
Which security AI model performs best?
The published numbers do not support a ranking. Microsoft reports the highest CyberGym figure, Sakana reports 86.9%, OpenAI reported 85.6% for GPT-5.5-Cyber, and Google declined to publish a figure at all. Because the results describe different configurations under undisclosed conditions, and none has been independently reproduced, they rank marketing confidence rather than capability.
Can I get access to these models?
Access is restricted across the category. Gemini 3.5 Flash Cyber is limited to governments and trusted partners. GPT-5.5-Cyber is in limited preview for critical-infrastructure defenders. MAI-Cyber-1-Flash enters public preview on 3 August. Fugu-Cyber is the most openly available, requiring only Sakana’s Token Plan. The gating reflects genuine dual-use concern rather than artificial scarcity.
Who is independently evaluating AI cyber capability?
The UK AI Security Institute and the US Center for AI Standards and Innovation both publish evaluations, including a joint assessment of Kimi K3 in July 2026 and the UK institute’s assessment of Claude Mythos Preview in April. Their reports disclose attempt counts, averages and failures rather than a single headline figure. NIST and CAISI have also signed agreements for pre-deployment cyber evaluation of frontier models from Google, Microsoft and xAI. None of this work uses CyberGym.
Update (2026-08-01): this article was expanded.
A companion piece on the benchmark problem was written separately and has been merged into this article rather than published alongside it, since both covered the same subject under the same focus keyphrase. This version adds the scoring-criterion discrepancy inside Microsoft’s own figures, the Berkeley leaderboard’s measured ceiling and its authors’ statement that no comprehensive benchmark yet exists, the provenance of the 85.6 comparison figure, the structural reasons vendors withhold methodology, and the behavioural evidence that models are finding real vulnerabilities regardless of what the scores say. Nothing in the original version was withdrawn or corrected.
