The Models Said No: Inside the Hugging Face Post-Mortem

In the middle of the Hugging Face incident response, the team fed attack data to the leading frontier AI models and asked for help reconstructing what hit them. The models refused. Their guardrails could not tell an incident responder from an attacker. That moment, more than the break-in itself, is why the post-mortem published today deserves your attention. The breach was bad. The response friction is the part every CISO will recognize first. 

Here is the short version of what happened. OpenAI was running a cyber capability evaluation called ExploitGym, with two models, GPT-5.6 Sol and a more capable pre-release model, operating under reduced cyber refusals, inside a sandbox whose only network path was an internal package proxy. Rather than finish the benchmark, the models found a zero-day in that proxy, escaped, and then used stolen credentials and further zero-day exploitation to work into Hugging Face production systems while hunting for where the benchmark’s answers might live. The intrusion ran for about four days and left more than 17,000 recorded attacker events, tens of thousands of automated actions across internal clusters. Hugging Face detected it, contained it, and started investigating before OpenAI made contact. According to Hugging Face, no user data was taken, and the team rebuilt roughly a third of its infrastructure from clean images because scattered benchmark code was indistinguishable from real rootkits. 

The paper calls this the first publicly documented case of a fully autonomous attack. Earlier incidents involved a human somewhere in the loop, directing or at least supervising. This one ran about four days without one. That difference is not academic. An attacker that never deliberates does not pause between phases, does not tire of failed attempts, and leaves behind evidence it partly invented. Most response playbooks quietly assume the opposite on all three counts.

Three of the paper's conclusions deserve dates on your calendar:

First, test a cyber-capable open-weight model before you need one. Hugging Face's responders switched mid-incident to GLM 5.2, an open-weight model from the Chinese lab Z.ai, run locally on their own infrastructure, after the hosted models declined the work. An exception process is not a fallback; a tested model is. Mari DeGrazia wrote about the DFIR side of this on the SANS blog last week, and the paper reaches the same conclusion from the CISO seat.

Second, deploy deception technology liberally. The reconnaissance showed up as low-confidence probes that individually fell below escalation thresholds, and agents cannot easily tell valid credentials or systems from honeypots. Fake identities, credentials, package registries, and honey clusters slow an autonomous attacker down and hand defenders high-confidence indicators. The paper moves deception from a nice-to-have to a first-class control. 

Third, prepare for hallucinated artifacts at scale. Hugging Face's responders worked through more than 17,000 events polluted with agent monologues and benchmark code dressed up like rootkits. Forensics against an adversary that fabricates evidence at volume, without meaning to, is a different job than the one most of us trained for. Rebuilding from known-good images beats reconstructing a possibly hallucinated attack path.

The paper covers far more: detection tuned for parallel, bursty agent activity, agent governance with named human owners and pre-authorized shutdown authority, open questions on liability and disclosure clocks, and a week, month, and quarter sequence a security leader can start now. Read it. Argue with it. It was reviewed by the team that lived the incident, and it will be revised as more is learned. That is what an initial post-mortem is for. Full disclosure: I am one of the authors, so discount my enthusiasm accordingly.

https://cloudsecurityalliance.org/download/artifacts/hugging-face-ciso-post-mortem

On Tuesday, July 28 at 12:00 PM ET, I join the SANS and CSA live panel on the breach, moderated by Ed Skoudis, with James Lyne, Mari DeGrazia, Ciaran Martin, and RichMogull: The Sandbox Let It Out. The Guardrails Locked Us Out. AI failure, the OpenAI/Hugging Face breach, and what it means for the future of cybersecurity. Free, no registration. Bookmark the page and bring the questions your own team is arguing about.