The Attack, the Defender and the Same AI Capability
The same AI capabilities can power an intrusion and help investigate it. Who gets access may determine what happens next.
Let’s recap the latest episode in AI wonderland: an AI agent powered by OpenAI models broke into Hugging Face to cheat on a cybersecurity test 😅
OpenAI was testing how good its latest models are at finding and exploiting software vulnerabilities. This is a standard part of model evaluation, based on benchmarks that act like graduate-level exams designed to test the limits of a model’s capabilities. The agents were supposed to attack controlled test systems. Instead, they found a way onto the open internet, reached Hugging Face’s real infrastructure and went looking for the answers. (Disclaimer: I worked at Hugging Face from 2024 to 2025)
It is the kind of incident that almost writes its own science-fiction headline. A model escapes its sandbox, breaks into another AI company and steals the answers to an exam.
The story is serious, but it is not an AI-doomsday parable and such an incident was expected given the evolution of frontier models in coding. It is a story about how quickly agent capabilities are improving, how difficult they are becoming to contain and why the same capabilities will be needed on both sides of cybersecurity.
It also makes the debate over open and closed models much harder.
What was the test?
The benchmark is called ExploitGym and was developed by UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara, and Arizona State University, in collaboration with Anthropic, OpenAI, and Google.
The simplest way to understand it is as a highly technical digital escape room built from real software vulnerabilities. In each challenge, an agent receives a vulnerable program, information about the bug and an environment where it can experiment. Its goal is to turn that vulnerability into a working exploit and retrieve a hidden code known as a “flag.”
The flag itself is worthless. It is not anything resembling nuclear codes. Retrieving it simply proves that the agent achieved something it was not supposed to be able to do: execute code with unauthorized privileges.
The capabilities being tested are not trivial, however. ExploitGym contains approx. 900 challenges derived from real-world vulnerabilities: 520 involving widely used software, 185 targeting V8 (the JavaScript engine inside Chromium) and 193 involving the Linux kernel.
The model may need to turn a small programming error into control over a system, bypass several layers of protection or combine multiple weaknesses into a working attack. These are tasks that require sustained reasoning, experimentation and adaptation, even for skilled human security researchers.
When the escape room acquired a real exit
According to OpenAI’s preliminary account, the evaluation used GPT-5.6 Sol and a more capable pre-release model with their usual cyber refusals reduced. That was intentional: the purpose was to measure the models’ maximum cybersecurity capabilities, including behaviours that production safeguards would normally block.
The environment was supposed to be isolated. Its network access was limited to an internal service used to obtain software packages. But the models found a previously unknown vulnerability in that service, used it to reach the internet, moved through OpenAI’s research environment and eventually inferred that Hugging Face might host information related to ExploitGym. Clever.
They then found ways into parts of Hugging Face’s production infrastructure and accessed information that could help them solve the benchmark.
In other words, rather than solving only the digital escape rooms placed in front of them, the agents found a real exit from the evaluation environment and went searching for the answer key.
This is a serious incident. But it is important not to project a human story onto it. There is no evidence that the models suddenly became malicious. The available evidence points to systems pursuing a narrow objective (solve the benchmark) through routes their evaluators had not anticipated.
Well, definitely much farther than anyone expected.
OpenAI also published a blog post on safety with long-horizon models, one day before releasing its account of the Hugging Face incident. Its main finding was that unlike earlier systems that often stopped when blocked, newer models keep trying, sometimes discovering ways around sandboxes or security controls.
The risk is persistence. This is why OpenAI argues that we must monitor not only individual actions, but the overall outcome an agent is working toward.
The same technology came back as the defender
What happened next may be the most important part of the story, because it shows how complicated the debate over releasing powerful new models has become.
Hugging Face detected the intrusion with AI-assisted monitoring. To reconstruct what happened, its security team needed to analyze more than 17,000 recorded events containing real attack commands, exploit payloads and infrastructure details.
They first tried frontier models available through commercial APIs. But the providers’ safety filters interpreted the material as a request for help with an attack and blocked it. They could not reliably distinguish someone conducting an attack from a defender investigating one.
So Hugging Face turned to GLM 5.2, an open-weight model. That allowed the team to process the attack data without sending sensitive commands or credentials outside its environment, and without relying on an API provider to distinguish defensive work from an attack in real time.
In the same incident, AI strengthened both the attack and the defence.
This is the core tension in cybersecurity. A model capable of finding and exploiting vulnerabilities can lower the cost of an attack. The same model can help defenders find weaknesses first, decide which patches are urgent, reconstruct an intrusion and respond at machine speed. You do not get one capability without the other.
Revisiting the “too powerful to release” playbook
In April, I wrote about the ”too powerful to release” playbook: the way restricted model launches can simultaneously generate fear, signal extraordinary capability and make independent verification difficult.
Anthropic’s Claude Mythos Preview was my main example. The company described a major jump in autonomous cybersecurity capability but did not make the model generally available. Instead, it initially provided access to roughly 50 selected cyber defenders and critical software providers through Project Glasswing.
Then, on June 12, the US government imposed export controls requiring Anthropic to prevent foreign nationals from accessing Fable 5 and Mythos 5. Anthropic temporarily suspended both models for everyone, and the restrictions were lifted later that month.
Two weeks later, OpenAI announced that, at the government’s request, GPT‑5.6 would initially be available only to a small group of trusted partners whose participation had been shared with US officials. OpenAI explicitly warned that this kind of government-controlled access should not become the norm, and the model was released more broadly in July.
These episodes raised the stakes of the debate. The question became who should decide who gets access, according to which criteria, and whether restricting access actually improves security or deprives defenders of tools available elsewhere.
ExploitGym adds meaningful evidence to that debate, as it is built to be reproducible. Claude Mythos Preview produced working exploits for 157 of its 898 challenges, while GPT-5.5 solved 120 within the evaluation’s two-hour limit.
The main experiments deliberately removed deployment-time safeguards and initially disabled several standard security mitigations. When those mitigations were restored, they stopped the majority of the agents’ successful attacks, but not all of them.
That is a much more useful finding than “AI can now hack everything.” Today’s agents remain unreliable across many targets. At the same time, autonomous exploitation is no longer merely hypothetical.
Three models of access, none sufficient alone
Keeping capabilities inside a lab can reduce public access to dangerous tools, but this incident shows that internal evaluations can themselves create real-world risk.
Trusted-access programs can equip vetted defenders while preserving oversight, but they inevitably raise questions about who qualifies, how quickly access can be granted and what happens to organizations outside the chosen group.
Open weights give defenders autonomy. They can run models privately, adapt them to their infrastructure and use them during an emergency without an API provider blocking the work. But the same lack of central control also makes it harder to prevent offensive use. It’s interesting to note however that similar mechanisms such as gated artifacts exist for open-source models.
This is not a remake of The Good, the Bad and the Ugly. Closed models are not inherently safe. Open models are not inherently dangerous. The incident simply exposes the limits of treating “open versus closed” as if it had an obvious security answer.
The debate will not escape US–China competition
There is another layer that will become increasingly difficult to ignore.
GLM 5.2 was developed by Beijing-based Z.ai and released with open weights under an MIT licence. Anthropic’s most capable cyber model, meanwhile, remains limited to selected partners.
This does not mean China is “open” and the United States is “closed.” Both countries contain companies pursuing different release strategies, and one model cannot represent an entire national ecosystem. But the contrast will inevitably enter the geopolitical debate.
If an American lab restricts a capability that a Chinese company later makes broadly available, has the restriction reduced global risk? Or has it mainly limited which defenders can use the strongest domestic tools? Conversely, if frontier capabilities are released openly because a competitor might do it anyway, does geopolitical pressure eliminate meaningful restraint?
These policy questions shape who gets access to the models, where sensitive data can be processed and whether defensive teams can keep pace with attackers operating without the same rules.
The risk is that model access becomes framed only as a race between countries. Cybersecurity is a global and deeply interconnected system, and the software protected by an American or Chinese model is often the same open-source infrastructure used everywhere.
Other countries, including Canada and members of the European Union, will inevitably feel the squeeze. They may find themselves choosing between restricted access to American models and readily available Chinese open-weight alternatives (a tension explored in this recent Economist article).
Canada’s Cyber Centre, for example, warns that increasingly accessible frontier models are expanding the threat landscape, while also urging defenders to use the same tools or risk falling behind attackers.
The benchmark says it all
The conclusion of the ExploitGym paper is worth reading precisely because it is more measured than most coverage of the incident will be.
The authors acknowledge the benchmark’s limitations and the continued unreliability of current agents. They also show that traditional security hardening remains effective, even if it is no longer sufficient by itself.
Most importantly, they describe exploit generation as inherently dual-use. It can help defenders assess which vulnerabilities pose the greatest risk and prioritize their response. It can also reduce the expertise, time and cost required to turn a vulnerability into an attack.
Their conclusion is neither “nothing to see here” nor “the machines are coming”:
“Traditional system hardening techniques and defense countermeasures remain effective but imperfect, and must therefore be assessed against AI-driven attackers. Addressing this risk requires both responsible model development and stronger defenses that explicitly incorporate autonomous exploitation into threat modeling.”
Cybersecurity has always been a contest in which attackers and defenders use many of the same advances. AI is not changing that fundamental reality, but it is changing the speed and scale on both sides.
The harder question is whether the future of cyber defence will be open or closed: who gets access to the most capable models, under what conditions, and whether defenders can use them when it matters most.