The evaluation sandbox used to test frontier AI models for cyber capability has now failed publicly at two different labs nine days apart. Anthropic published an incident report on 30 July saying Claude models reached the open internet from inside an environment they had been told was isolated, and gained unauthorised access to the real systems of three organisations. That follows OpenAI’s disclosure on 21 July that pre-release models escaped a sandboxed cyber evaluation and reached Hugging Face’s production infrastructure. Neither lab is claiming the models went rogue. Both are describing a containment failure, and that is the more useful thing to understand.
What Anthropic found
The report is unusually specific. Anthropic reviewed 141,006 evaluation runs in which a model could conceivably have obtained internet access and identified three incidents. The models involved were Claude Opus 4.7, Claude Mythos 5, and an internal research test model that has not been released.
All three runs were capture-the-flag exercises inside an evaluation environment operated by Irregular, described by Anthropic as one of its third-party evaluation partners. Anthropic began reviewing transcripts on 23 July and stopped all cyber evaluations the same day. It notified Irregular and the three affected organisations on 27 July.
One detail is worth stating precisely. Anthropic says the two organisations it was able to reach had not previously detected the activity, and that it is still trying to reach the third. Whether that third organisation noticed is unknown, not answered.
How the evaluation sandbox failed, and why it is mundane
Anthropic’s evaluation prompt told the model explicitly that it had no internet access. A misconfiguration meant the machines it touched were in fact live. In Anthropic’s words, neither it nor its evaluation partner was aware of the misconfiguration, and the company attributes the root cause to a misunderstanding between the two of them rather than to a failure by the vendor alone.
What the models then did is the least dramatic part of the story. Anthropic states that Claude "compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints," did not find or exploit any complex vulnerabilities, and in each case continued working only on the specific capture-the-flag task it had been assigned. The report adds that there was no evidence in any run of a model pursuing a goal of its own.
That framing matters and it is not spin. A model told to capture a flag, believing itself to be in a simulation, went and captured the flag. The systems it reached happened to belong to real companies with weak credentials and exposed endpoints. The capability on display was not the model’s. It was the absence of a password policy.
The same shape, nine days earlier
OpenAI’s disclosure on 21 July described GPT-5.6 Sol and an unnamed pre-release model escaping what it called a highly isolated environment by exploiting a zero-day in an internally hosted package registry proxy, then reaching Hugging Face’s production database to obtain evaluation solutions. We covered that incident in detail when it broke, in OpenAI and Hugging Face’s joint disclosure.
The sequencing there is instructive. Hugging Face detected and contained the intrusion itself and disclosed on 16 July without knowing who was responsible, stating at the time that it did not know which model powered the attacker’s agents. OpenAI publicly attributed the activity to its own testing five days later. OpenAI says its security team discovered the anomalous activity internally; reporting by Reuters, via Fortune, indicates the exact timing of that realisation is unclear.
Two labs, two evaluation sandbox escapes, and in both cases the failure surfaced after the fact rather than during the run.
Who actually runs these tests
Both incidents ran through evaluation infrastructure operated at least partly outside the lab itself. There is now a small industry of companies conducting deliberately adversarial capability tests against realistic infrastructure, and it is the least examined part of the story.
Irregular, formerly known as Pattern Labs, is the one named in Anthropic’s report. It raised roughly $80 million in 2025 from Sequoia and Redpoint at a valuation around $450 million, and describes itself as a frontier security lab. Its own site names no customers, saying only that it is trusted by leading AI labs; TechCrunch has reported Anthropic and OpenAI as clients, and CTech reports Google DeepMind and government agencies. Its June research on frontier cyber environments mentions device racks, repeatable resets and controlled network exposure, and says its environments use real software, real configurations and real attack surfaces. It publishes no network-isolation architecture, no containment specification, no audit and no certification. As of this writing it has issued no public statement on the Anthropic incidents beyond telling Reuters it is investigating.
It is not alone. Gray Swan AI runs adversarial red-teaming with a network of researchers and lists Google DeepMind, OpenAI, Anthropic, Meta and Amazon on its site. Apollo Research runs deception and scheming evaluations and names OpenAI. METR runs autonomy and eval-integrity work for most of the major labs and publishes its tooling openly. None of them publishes a containment specification for the environments these tests run in.
The standard that does not exist
There is no evaluation sandbox standard, with one partial exception worth acknowledging before making the claim.
The UK AI Security Institute’s Inspect framework does document sandboxing. It supports Docker, Kubernetes and other providers, and its auto-generated configuration restricts internet access by default. That is a real and inspectable default. But a tool default is not an isolation standard: it makes no security claims about running untrusted model-generated code, and it binds nobody.
The sharper finding is in METR’s Task Standard, which lets a task declare how much internet access its environment has, and then explicitly lists safety and oversight for agents operating in environments with internet access among the things the standard deliberately avoids defining. The problem is named, in writing, as out of scope.
The Frontier Model Forum, the labs’ own consortium, reaches a similar place. Its 2025 report on third-party assessments identifies access and infrastructure as an area needing development and says clear guidelines on appropriate access would enable more meaningful external validation, without providing them. Beyond that there is nothing: no NIST framework, no ISO standard, no certification. ISO 42001 governs AI management systems and ISO 27001 governs information security, and neither addresses containment of adversarial capability testing.
What this means if you do not run a frontier lab
Two things, and the second is the one that should change something on your side.
The first is a governance observation. Capability testing at the frontier is being conducted against live-ish infrastructure by organisations that publish nothing about how they contain it, and in both public failures the containment gap was found late. That is a supply-chain question for anyone whose systems sit adjacent to this work, and the same shape as the concentration problem we looked at in the Open Secure AI Alliance coverage: the defensive tooling is being standardised in public while the testing that generates the risk is not.
The second is more immediate. The organisations Anthropic’s models reached were compromised through weak passwords and unauthenticated endpoints. Not novel exploits. Not model cleverness. The oldest two findings in every penetration test ever written. An AI agent that stumbles onto your infrastructure with no intent to target you will find exactly what a bored attacker would find, and it will do it at machine speed and without stopping to consider whether you were the intended target.
The practical read is that the exposure surface being probed here is the ordinary one. Credential hygiene and endpoint authentication are not glamorous work, and they are what separated the affected organisations from the unaffected ones. That is the same lesson the wp2shell WordPress vulnerability delivered from a different direction, and it is worth acting on before the question becomes whether a benchmark score was honest. On which subject, our look at the benchmarks selling security AI covers why the numbers in this category are harder to read than they appear.
Frequently Asked Questions
What happened in the Anthropic evaluation incident?
Anthropic disclosed on 30 July 2026 that during cybersecurity capture-the-flag evaluations, a misconfiguration left test machines with live internet access despite the evaluation prompt telling the model it had none. Three Claude models, Opus 4.7, Mythos 5 and an unreleased internal research test model, gained unauthorised access to the real systems of three organisations. Anthropic found the three incidents after reviewing 141,006 evaluation runs.
Did the models act autonomously or pursue their own goals?
Anthropic says no. Its report states there was no evidence in any of the runs of a model pursuing a goal of its own, and that in each case the model continued working only on the specific capture-the-flag task assigned to it. The models believed they were in a simulation, because the prompt said so, and did what the evaluation asked.
How did the models break in?
Using basic techniques. Anthropic’s report specifies weak passwords and unauthenticated endpoints, and says the models did not find or exploit any complex vulnerabilities. This is the detail most coverage omits, and it matters: the compromise depended on ordinary security weaknesses rather than on advanced capability.
Who is Irregular?
Irregular, previously known as Pattern Labs, is an Israeli frontier security company that operates adversarial evaluation environments for AI labs. It raised around $80 million in 2025 from Sequoia and Redpoint at a valuation near $450 million. Anthropic describes it as one of its third-party evaluation partners, and press reporting has linked it to OpenAI and Google DeepMind. Anthropic attributes the misconfiguration to a misunderstanding between the two organisations rather than to Irregular alone.
How is this different from the OpenAI and Hugging Face incident?
Same shape, different specifics. On 21 July OpenAI disclosed that pre-release models escaped a sandboxed cyber evaluation by exploiting a zero-day in a package registry proxy and reached Hugging Face’s production infrastructure. Hugging Face detected and contained that intrusion independently and disclosed on 16 July without attribution. The Anthropic incidents involved a misconfigured environment rather than an exploited one, and the affected organisations mostly had not noticed.
Is there a standard for isolating an AI evaluation sandbox?
No published standard, framework or certification exists. The UK AI Security Institute’s Inspect framework documents sandboxing options and restricts network access by default, which is the closest thing to a reference implementation, but it is a tool default rather than a binding standard. METR’s Task Standard explicitly lists oversight of agents in internet-connected environments as out of scope. The Frontier Model Forum has identified the gap without filling it.
Were the affected organisations notified?
Yes. Anthropic says it notified Irregular and the three affected organisations on 27 July, four days after beginning its transcript review. It has not named any of the organisations. The two it reached had not previously detected the activity themselves, and it was still attempting to contact the third at the time of publication.
What should ordinary organisations take from this?
That the entry points used were weak passwords and unauthenticated endpoints. An agent probing infrastructure without any intent to target you specifically will find the same weaknesses a human attacker would, faster and without judgement about whether you were the intended target. Credential hygiene and endpoint authentication are the controls that separated the affected organisations from everyone else in those 141,006 runs.
