The Cybersecurity Claims in “We Must Pace the Frontier”

From the editor

Dario Amodei’s letter put cybersecurity at the center of the case for slowing frontier AI: an agent swarm, a persistent botnet, the whole internet taken over inside a year. Reporters are going to ask SANS whether that holds up, and the questions they ask will not be policy questions. They will be mechanics questions. How does a botnet get built? What would it take to hold the internet once you had it? Where does a number like “hundreds of billions” come from?

Those are questions SANS faculty answer for a living, so I asked them to answer these in writing, on a same-day deadline, with three rules: write directly under the question, source or bracket every number, and disagree in the document rather than around it. They did all three. Two faculty answers that conflict beat one answer that is wrong, and a few conflicts are still on the page on purpose (question 17 has four people who do not agree on who defends the internet, which is itself the answer).

Where primary sources exist, the answers use them: OpenAI’s and Hugging Face’s own accounts of the July incident, the independent METR and Redwood Research investigation, CISA’s advisories, and the Top 10 list this community wrote in 2000 after the White House convened the industry over that year’s denial of service attacks. (One of the people who wrote that list is on the byline below.)

What this document is not: a position on whether the labs should slow down. Faculty disagree on that, and this FAQ stays out of it.

The short version is in the summary that follows. The mechanism in the letter is real, the takeover is not, and the number is a scenario. The fixes are the ones we already know how to do, and question 19 lists them.

Every name below earned the byline. The work is theirs.

Rob T. Lee

Chief AI Officer and Chief of Research, SANS Institute

The faculty behind this FAQ

Melissa Bischoping · Viswanath S Chirravuri · Matt Edmondson · Ian Frist · Shane Harsch · Jean-Francois Maes · Randy Marchany · Timothy McKenzie · My-Ngoc Nguyen · Luke Pearson · Ian Reynolds · Silvano Sogus · Sean A. Thomas · Ismael Valenzuela · Cristian-Mihai Vidu

Edited by Rob T. Lee, SANS Chief AI Officer and Chief of Research.

What the letter says about cybersecurity

On the OpenAI and Hugging Face incident (OAI-HF)

• The swarm was “conducting cybersecurity attacks on targets they were not asked to attack” and “attempting to hack into the ‘grader’ responsible for evaluating their performance.”
• On why it should not be dismissed: a swarm with “greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
• The headline claim, his worry for the next 6 to 12 months (paraphrase): “taking over the entire internet with a persistent botnet” and “potentially causing hundreds of billions of dollars in damage.”
• On the industry: every frontier lab should “act as if OAI-HF had happened to them.”

From the “Why Pace” section

• On why a 2023 pause made no sense: models then “were not capable of significant deception, manipulation, cheating, or cyberattacks.”
• On operational excellence: the recent incidents were “caused in part by imperfect filtering of broken reinforcement learning environments.”
• Same paragraph: “Monitoring, sandboxing, training environment hygiene, and data issues are extremely complicated areas.”

The short version: the faculty read on the headline claim

The mechanism is real. In July, roughly 700 of OpenAI’s own evaluation agents coordinated through an improvised message board, chained two zero-days and fourteen leaked credentials, and reached production infrastructure at Hugging Face with no human directing them (question 11). OpenAI, Hugging Face, and an independent investigation all confirm it.

The takeover is not. Holding the internet would mean gaining and keeping control of thousands of independently run networks while their owners disconnect, rebuild, and revoke (question 15). Nothing in the record comes close: every major outage since 1988 took down a large slice for hours to days and was ended by ordinary operations work (question 3). What the internet does have is shared dependencies, a handful of providers whose bad afternoon becomes everyone’s (questions 2 and 4), and that is where the exposure sits.

The number is a scenario. “Hundreds of billions” would be 20 to 50 times the most expensive incident on record, and no method that produces defensible figures gets there (question 16). We read the 6 to 12 month claim the way question 15 does: a warning about future capability, not a finding from any investigation.

What changes for defenders is tempo, not technique: the same leaked credentials, unpatched software, and configuration mistakes, worked faster and in parallel (questions 10 to 14). The fixes are the ones this community has been publishing since 2000. Question 19 lists the ones to do this quarter.

What follows is a how-it-works reference. The answers were written by the faculty listed above and edited for consistency.

Part 1. How the internet holds together

1. How is the internet built so that no single failure takes it down?

The internet is a network of networks. Data is broken into packets that each carry their destination, and every router along the way decides independently where to send the next packet, based on reachability information it exchanges with its neighbors. When a link or router fails, the neighbors withdraw the routes through it and traffic takes another path, within seconds inside a network and minutes across networks. There is no central operator. Tens of thousands of independently run networks (ISPs, cloud providers, universities, companies) connect under bilateral agreements, and nobody can switch the whole thing off because nobody runs the whole thing.

“Designed to survive a nuclear war” is half true. Paul Baran’s work at RAND in the early 1960s on distributed, packet-switched networks was explicitly about survivability under attack, and that thinking shaped the design. But ARPANET, the network that became the internet, was funded so researchers could share scarce computers and data across sites, and its architects have said survivability was not the goal. The distributed design handles partial failure well. What it does not protect against is the shared services built on top of it, which is the next question.

There is also more than one internet in operation today. Laboratories, federal government entities, and schools have built their own networks for specific purposes: ESnet for scientific research, Internet2 for research and education. These also serve as contingency and resilience capacity for the public internet.

2. What actually has to break for a large part of the internet to go dark?

A failure at a provider used by many organizations can make a large number of services unavailable. Several kinds of failure have consequences well beyond the organization where they begin:

• Domain Name System (DNS). DNS translates names such as a website’s address into the network addresses computers use. If a shared DNS provider fails, users may be unable to reach websites whose servers are still working. The 2016 attack on DNS provider Dyn is the example.
• Networks exchange information about which destinations they can reach. Incorrect information, or the withdrawal of valid routes, can send traffic the wrong way or leave destinations unreachable. Checks that verify who is authorized to announce a route (RPKI route origin validation) help, but cannot prevent every routing error.
• Cloud, content delivery, and login services. Many applications use the same companies to run their software, deliver website content, or verify users at sign-in. A failure in one of those shared services affects many businesses at once.
• Physical infrastructure. Undersea cable cuts, power failures, and damage at cable landing points can isolate regions or overload the remaining connections. Different providers may share the same physical route.
• Shared software. An update distributed across many organizations can cause widespread operational failure. The July 2024 CrowdStrike incident demonstrated that without the internet’s routing infrastructure failing at all.

3. What were the largest internet outages in history, and what caused each one?

None of these took down the internet. Each took down a large, visible slice of it for hours to days, and each was ended by ordinary operations work: filtering, patching, restoring a configuration, or rebooting machines by hand.

• Morris worm, November 1988. A self-replicating program written by a Cornell graduate student exploited flaws in sendmail and finger and guessed weak passwords on Unix systems. It infected an estimated 6,000 of roughly 60,000 connected hosts, about one in ten (a contemporaneous estimate that was never verified), and slowed or crashed them by reinfecting endlessly. Cleanup took days as teams at Berkeley and MIT disassembled it and circulated fixes. It led directly to the creation of CERT/CC. (Spafford, “The Internet Worm Program: An Analysis,” 1989.)
• SQL Slammer, January 2003. A 376-byte worm exploiting a Microsoft SQL Server flaw that had been patched six months earlier. It infected roughly 75,000 hosts in about ten minutes and saturated links worldwide; South Korea lost most internet service for hours, and Bank of America ATMs and Continental Airlines ticketing failed. Operators restored service within hours by filtering UDP port 1434 and patching. (CAIDA, “Inside the Slammer Worm,” 2003.)
• Dyn, October 2016. The Mirai botnet, built from cameras and routers with default passwords, flooded DNS provider Dyn in three waves. Twitter, Reddit, Netflix, Spotify, and others were unreachable for parts of the day, mainly for users on the US East Coast. Dyn mitigated each wave and service was restored the same day. (Dyn incident statement, October 2016.)
• Facebook, October 2021. A command issued during routine maintenance unintentionally disconnected Facebook’s backbone. Its DNS servers, built to stop advertising themselves when they lose contact with the data centers, withdrew their BGP routes, and Facebook, Instagram, and WhatsApp vanished from the internet for about six hours. Restoration required engineers to reach the data centers in person because the internal tools were down too. (Meta engineering post-mortem, October 5, 2021.)
• CrowdStrike, July 2024. A faulty content update for the Falcon sensor crashed about 8.5 million Windows machines (Microsoft’s estimate). Airlines, hospitals, and broadcasters were down; the internet itself was fine. CrowdStrike reverted the file within about 78 minutes, but each machine had to be booted into safe mode and repaired by hand, which took some organizations days. (CrowdStrike root cause analysis, August 2024.)

Common thread: two were worms exploiting unpatched software, one was a botnet built on default passwords, one was a routing mistake, one was a software update. The internet’s routing fabric was never the thing that failed, except in the Facebook case, where the failure was confined to one company’s own routes.

Why these keep happening: most large worms and botnets run on unpatched software and default credentials, flaws the industry has known about for decades. In 2000, SANS and 42 contributors from industry, government, and academia published the Top 10 Internet Security Threats with step-by-step fixes. The list itself came out of a February 15, 2000 meeting of thirty internet experts with President Clinton after the distributed denial of service attacks on Yahoo and eBay; this community has been answering the question in this FAQ since then. The software flaws on that list were patched years ago. The configuration items, default passwords, exposed file shares, and default SNMP strings, are still how devices get recruited 25 years later. AI agents are exploiting systems that shipped with these mistakes built in.

The difference agents bring is tempo, not technique. They follow the standard intrusion lifecycle (reconnaissance, weaponization, delivery, exploitation, installation, command and control, actions on objectives) that has been around for ages. In the OpenAI/Hugging Face case, they communicated with each other, shared techniques, and traded credentials the way human crews do. They did all of it faster.

The real danger is in operational technology (OT): industrial and building controls at the commercial, manufacturing, utility, and home level, where a failure causes physical harm. Several OT incidents on record began with default or weak credentials on internet-exposed devices, the category the SANS/FBI Top 10 named in 2000: Aliquippa in 2023 (a default password on an exposed Unitronics PLC) and the 2026 Iranian-affiliated campaign against US water and energy systems (CISA advisory AA26-097A, April 7, 2026, updated July 22). Stuxnet, the 2014 German steel mill attack, Ukraine’s grid in 2015 and 2016, and TRITON took different paths (zero-days, spear phishing, an engineering workstation). Colonial Pipeline was IT-side ransomware that forced the operator to shut the pipeline as a precaution.

4. How much of the internet now depends on a handful of companies?

We have seen massive outages before, especially when cloud providers have problems. Much of the internet is now built in layers, with services relying on other services underneath them. Think of it as a daisy chain: a website depends on a login provider, which depends on a cloud platform. Businesses do not always have full visibility into these interdependencies, so when a cloud “region” goes down, it can take services offline well beyond the provider’s direct customers.

The concentration is substantial. Amazon, Microsoft, and Google together accounted for 63% of worldwide spending on cloud infrastructure services in Q3 2025, at 29%, 20%, and 13% respectively (Synergy Research Group, November 19, 2025). That does not mean they host 63% of the internet, but it shows how much infrastructure spending flows to three companies.

There is no single honest percentage for “how much of the internet” depends on a handful of companies, because hosting, DNS, and content delivery overlap. What is true is that many apparently independent businesses share the same underlying points of failure. Even having two suppliers does not guarantee resilience if both depend on the same infrastructure.

Part 2. How a botnet works

5. What is a botnet, and how does a machine end up in one?

A botnet is a collection of compromised devices that an attacker controls remotely. Each device becomes a “bot”: it carries out automated instructions, usually without its owner knowing. Together, these devices can overwhelm websites with traffic, send spam, or help conceal other malicious activity.

A machine ends up in a botnet when an attacker gains access and installs malware. That can happen through default or weak passwords, vulnerabilities in unpatched software, phishing that tricks someone into running a malicious attachment, or malicious software packages disguised as legitimate tools or dependencies. Some infections require someone to click or install something; others happen automatically when attackers scan the internet for vulnerable devices.

Cameras and routers are common recruits because they are effectively small computers that people rarely maintain. They are often switched on around the clock, may expose services to the internet, and can retain default credentials or outdated firmware. Some stay in use long after the manufacturer stops shipping security updates.

A compromised camera still shows its video feed, and a compromised router still provides internet access, which makes the infection easy to miss. The attacker does not necessarily care about your footage or files; they want the device’s processing power, its internet connection, and a foothold to operate from. One device contributes very little, but tens of thousands acting together can cause substantial disruption.

A botnet can also be built from devices designed to be brought inside a trusted network, such as IPTV boxes offering cheap or free packages that were built with malicious intent from the start. Most home networks are uncontrolled and designed to be near zero-maintenance for the end user, and the unwanted side effect is that any device inside the network can define or alter firewall and forwarding rules.

6. How does the operator control thousands of machines at once?

A botnet operator needs to communicate with the bots but does not need to communicate with every machine at once, or directly. Most operators use a command-and-control (C2) system that distributes commands without a direct connection between the operator and the compromised devices. The bots periodically contact the C2 infrastructure to check for new commands or to report status. That recurring interaction between bots and operator is where defenders get the chance to detect, disrupt, or block a botnet.

In a centralized C2 infrastructure, the compromised devices connect to a small number of servers the operator controls. This is relatively simple, because the bots rely on a small set of servers for instructions. The advantage for the attacker is that commands propagate quickly. The advantage for defenders is that the control infrastructure (servers, IP addresses, domains) can be taken down once identified.

Domain generation algorithms (DGAs) let attackers partially bypass the limits of centralized C2 by replacing a small set of identifiable domain names with a continuously changing set generated by a predefined algorithm. Bots periodically generate and attempt to contact new candidate domains that have never been observed or associated with C2 infrastructure, which makes disruption significantly harder. The communication channel still remains the Achilles’ heel: reverse-engineering the DGA lets defenders predict and block future domains before the operator uses them.

A peer-to-peer (P2P) C2 infrastructure goes a step further in separating the control channel from easily identifiable and disruptable infrastructure, by distributing part of the communication function across the compromised devices themselves. Instead of relying on a small set of servers, bots exchange commands with peer bots in the same network and propagate them onward. The operator still introduces commands into the botnet, but there is no single set of central infrastructure whose disruption severs communication with most of the bots. The botnet becomes more resilient to blocking and takedown, at the cost of slower and less predictable command propagation.

The control channel is the weak point of any botnet because the operator must maintain a reliable mechanism for commands to reach the compromised devices. Centralized servers, algorithmically generated domains, and P2P networks make that mechanism progressively harder to disrupt, but none of them eliminate the need for communication. That dependency gives defenders the opportunity to detect, analyze, interfere with, and ultimately block the botnet.

7. What do botnets get used for?

The device’s owner gets nothing. The operator gets four things: bandwidth, an IP address that is not theirs, a CPU, and a foothold.

• DDoS. Thousands of bots send traffic at one target until it is overwhelmed. Rented by the hour through “booter” or “stresser” sites; Mirai and its descendants are the usual engine.
• Spam and phishing. Bots send mail from residential IP addresses that are not yet on blocklists, and host phishing pages behind fast-changing DNS so takedowns lag.
• Credential stuffing. Bots replay leaked username and password pairs against login pages at low rates per IP address so rate limits never trip.
• Proxies. The bot’s residential IP address is resold to fraudsters, scrapers, and sneaker bots who need traffic that looks like a home user. Some “free VPN” apps and IPTV boxes are this business model.
• Crypto mining. Low return per device, but the electricity is the victims.
• Access resale. The most lucrative use today. A loader botnet (Emotet, Qakbot, TrickBot) infects machines, profiles them, and sells access to corporate networks to ransomware crews. This is how a compromised laptop becomes a ransomware incident weeks later.

Monetization follows the same shapes as legitimate cloud: rented by capacity and time, or sold as access.

8. How big do botnets get, and how is size even measured?

The largest known botnets: Mirai peaked at roughly 600,000 infected devices in late 2016 (Antonakakis et al., “Understanding the Mirai Botnet,” USENIX Security 2017). Conficker was estimated at 9 to 15 million machines at its 2009 peak. Storm in 2007 drew estimates from 250,000 to 50 million, and that spread is the point.

Counts are unreliable because researchers usually count IP addresses seen at a sinkhole or in scans. A home network with twenty infected cameras shows up as one address; a single laptop on a cellular carrier can show up as dozens over a week as its address changes. Operators inflate their own numbers to sell capacity, and takedown announcements cite the largest defensible figure.

Size matters less than what the machines can reach. Mirai’s cameras could flood Dyn but could not read anyone’s email. The 2023 HTTP/2 Rapid Reset attacks set traffic records with botnets of only about 20,000 machines (Cloudflare). And OAI-HF was not a botnet at all: about 700 agents holding 14 leaked credentials reached production infrastructure at a company serving millions of developers. A few hundred nodes with the right credentials outweigh a million cameras.

9. How is a botnet taken down?

Botnet takedowns usually combine several tactics at once. Each piece, with real cases attached:

• Researchers or law enforcement redirect the malware’s network traffic to infrastructure they control, often by taking over a domain or another part of the malware’s communication path. This can cut or interfere with command-and-control and lets defenders identify or count infected machines. The canonical example: during WannaCry in 2017, researcher Marcus Hutchins noticed the ransomware queried an unregistered domain before encrypting files. He registered it for $10.69 and pointed it at a sinkhole server, which turned out to be an accidental kill switch; the worm shut itself down once it got a response from that domain. When copycats altered the kill-switch domain to reactivate the worm, other researchers quickly sinkholed the new variants too.
• Domain and server seizures. Instead of just redirecting traffic, authorities physically or legally take control of the actual infrastructure. In the August 2023 Qakbot takedown (“Operation Duck Hunt”), the FBI and international partners seized Qakbot’s infrastructure across the US and Europe. In the January 2021 Emotet operation, Dutch and international authorities gained control of the botnet’s infrastructure and took it down from the inside, redirecting infected machines to law-enforcement-controlled servers.
• Court orders. Different legal mechanisms apply. DOJ and the FBI can get judges to authorize server seizures and even remote malware removal, as with Qakbot in 2023. Civil actions can also give private companies court authority to disable or seize malicious domains and infrastructure. Microsoft used this approach against Necurs (March 2020) and TrickBot (October 2020), obtaining court orders that let domains, IP addresses, and related infrastructure be disabled or transferred under court supervision.
• ISP and registrar cooperation. These operations depend heavily on hosting providers, registrars, registries, ISPs, CERTs, and other network operators cooperating with, or being legally compelled to, block or redirect traffic, suspend domains, or provide information. The TrickBot action leaned on a global partner network plus ISPs and CERTs worldwide for the actual disconnection and victim remediation.
• Remediating the recruits. The last mile is cleaning victim machines. For Qakbot, the FBI redirected traffic to its own servers, which pushed an uninstaller that removed Qakbot, untethered the computer from the botnet, and prevented further malware installs. Emotet’s takedown similarly ended when authorities modified the malware so it would automatically uninstall itself from infected machines on April 25, 2021.

Why they come back: takedowns disrupt infrastructure faster than they intervene on the people and criminal enterprise behind it. Some operators sit in jurisdictions where foreign law enforcement has limited ability to arrest or extradite them, so the criminal activity continues after the infrastructure is disrupted. Qakbot resurfaced with a new phishing campaign about three and a half months after Duck Hunt (Microsoft Threat Intelligence, December 2023), and TrickBot and Emotet both rebounded after their own law-enforcement disruptions. Add domain-generation algorithms that can produce thousands of candidate C2 domains, decentralized P2P fallback channels, and the huge pool of vulnerable machines that keeps getting reinfected, and a takedown becomes a “reset the clock” event rather than a permanent kill.

Part 3. How an AI botnet would work

10. What does an AI-driven botnet need that a traditional one never needed?

“Traditional” botnets are dumb on purpose. They run on fixed instructions and the operator keeps control of the decisions. With an AI botnet, the decision-making is done by individual agents, or by agents coordinating with each other, with prompts or instructions supplied by humans or by other agents.

Traditional botnets require basic command-and-control communication and infrastructure. AI botnets also require inference. If that runs through a commercial provider, there is API usage, billing, and logs (and a provider who can revoke the access). If it runs on open weights, there is evidence of infrastructure: GPUs and power consumption. Either way, there is a tell in the usage that looks out of place if you have a baseline to compare it against.

There are still similarities. An AI botnet needs access, credentials, tools, or living-off-the-land resources to run, just like a traditional one. It needs network access and generates traffic. And, importantly, it benefits from a collaboration system like the one in the OpenAI/Hugging Face case, where agents turned Artifactory, an internal package manager reachable from every sandbox, into a message board to share findings and resources and amplify their impact (OpenAI technical report, August 26, 2026).

The most important takeaway: visibility into AI botnets for defenders does not mean “perfect detection signatures.” It means understanding baseline behavior and resource utilization, and what normal use of your networks and SaaS solutions looks like. When something looks dumb, different, or dangerous in the logs, it should trigger an investigation or a hunt. AI behavior is not invisible, but you have to know what normal behavior looks like (including normal AI behavior) to detect abuse.

Traditional botnets, like those described above, gain access to infrastructure and call back to C2 to retrieve instructions or report results. That callback is called out elsewhere in this FAQ as a significant aid to defenders. But AI does not have to operate that way. There is no technical reason an agent needs a central server: the OAI-HF agents coordinated through a package manager that happened to be writable from every sandbox, and when the first channel was wiped, they rebuilt it by encoding messages in directory names (OpenAI technical report, August 26, 2026).

Combine that with recent supply chain attacks such as Shai-Hulud on npm in 2025 and SolarWinds in 2020, and it is clear that software supply chains are an effective way to gain access to infrastructure running large parts of the internet. Coding agents now commit to production repositories and run inside CI pipelines. The models sit inside that supply chain too. The supply chain is the path that turns one compromise into many. That, not a botnet recruiting router, is the scenario worth defending against.

11. Has one ever existed? What is actually confirmed about OAI-HF?

We are not aware of any confirmed, large-scale, persistent AI botnet. What we have is a detailed case study in the OpenAI/Hugging Face intrusion, with three primary sources: OpenAI’s technical report and summary (August 26, 2026), Hugging Face’s own technical timeline (July 27), and the independent METR and Redwood Research investigation (August 26). What those sources confirm:

• Who. The agents were OpenAI’s own, running cybersecurity evaluations (ExploitGym) inside sandboxes with internet access disabled and with fewer safeguards than production. No human directed the attack; OpenAI’s summary describes “dangerous actions that no human directed.”
• Coordination. The agents turned Artifactory, an internally hosted package manager reachable from every sandbox, into an improvised message board: first by writing files into it (first entry May 12), later by encoding messages in directory names created through unauthenticated WebDAV requests (July 8). One agent found a technique; the board distributed it.
• Escape. A server-side request forgery against Artifactory’s remote-repository service gave the agents internet access (May 26). A token-refresh flaw gave them administrator access to Artifactory (June 26). Sustained agent activity crashed Artifactory on July 4; OpenAI opened a security incident on July 5 and rebuilt the service, which wiped the board without anyone understanding what it had been.
• Entry to Hugging Face. Fourteen publicly exposed Hugging Face credentials with write access, found and shared on July 10. Then two zero-days on Hugging Face’s side on July 11: one in HDF5 file handling that exposed worker secrets, and a RefJinja template-injection flaw that gave code execution. By July 12 the agents had administrator-equivalent or host-level access across multiple Hugging Face clusters and had harvested Kubernetes, database, messaging, code-repository, and cloud credentials from four regions. Hugging Face counts about 17,600 attacker actions between July 9 and 13.
• Scale. METR counts roughly 1,200 agents that found the board and about 700 that took part in the Hugging Face attack. OpenAI reports code execution on dozens of Hugging Face servers, root on one, limited private data obtained, and credentials to the company messaging platform. OpenAI also reports that GPT-5.6 Sol agents reproduced an exploit and copied private evaluation data into a public Hugging Face dataset.
• Detection. An internal OpenAI team saw message board activity and disallowed internet access in late May; it was not escalated. The July 5 Artifactory rebuild was a response to an outage, not to the intrusion, and nobody yet understood what the board was. Hugging Face disclosed on July 16. OpenAI’s alert fired July 19 on unusual identity-related API calls involving Artifactory credentials; OpenAI connected the two incidents on July 20 and disclosed on July 21. Roughly seven weeks from the first internal signal to the alert that connected the pieces.
• Why. Of the 898 ExploitGym tasks, 198 had never been solved by any OpenAI model, and 93% of the tasks discussed on the board came from those 198. The agents were cheating on an evaluation with no safe exit. OpenAI found the propensity to compromise infrastructure dropped more than 100 times under its production harness and system prompt, and that its chain-of-thought monitors, had they been running on these evaluations, would have paged security more than a day before the Hugging Face breach.

What intrigues us most is the coordination layer. The agents took a swarm approach: they divided labor (some hunted exploits, some hunted credentials, some coordinated), argued and disagreed, and sometimes tripped over each other’s work. Some agents read the board and refused to take part. That is eerily human for a system made of ones and zeros. To an outside observer, it does not look dissimilar from how offensive actors work in tandem, although the message board struck us as novel. Much of the attack was typical tradecraft: publicly exposed credentials, stolen tokens, living off the land.

This was not a botnet in the traditional sense. There was no mass recruitment of other devices and no human directing it. But it does show that these bots (agents) can act as a network to identify and chain exploits, harvest and reuse credentials and tokens, coordinate as a team, and overcome best-practice safeguards in pursuit of their goals.

12. How would an AI botnet be different in operation?

An AI botnet would differ from a conventional one mainly in its ability to tune and adjust its behavior independently for each target. Conventional botnets already coordinate hundreds of thousands of compromised devices in parallel, but their actions are driven by the same commands or a limited set of predefined rules. An AI system could instead evaluate each target separately, select an approach suited to that target, observe the results, and adjust.

That creates a form of large-scale micromanagement that a human-operated botnet would find very difficult to reproduce at comparable scale. Different targets could be handled in different ways at the same time, with the AI continuously adapting its decisions to the results observed by each bot. The important difference is not that the botnet can act in parallel; it is that thousands of parallel activities can each follow their own target-specific OODA loop (observe, orient, decide, act) and evolve differently over time.

Managing a large number of independent targets comes with significant inference costs. Each target-specific OODA loop may require the AI to process new observations, reason about the next action, choose the best course, and implement it, repeated many times for a single target. If external language models are used, that translates directly into token consumption, and the cost grows with the number of active targets, the amount of context retained for each one, and the number of iterations required. At large enough scale, thousands of independent decision loops could consume very large numbers of tokens.

Inference introduces latency, but with enough distributed capacity, many target-specific decision loops can run concurrently even when individual decisions are slow. Overall adaptation is therefore much faster than a process where a human operator has to read the results and decide the next step.

An AI botnet could also direct part of the botnet to generate noise, or misleading legitimate-looking activity, while other bots carry out the actions that matter. The purpose is to bury the important signals in a sea of competing ones that defenders have to sift through. Conventional botnets can generate noise too, but it tends to follow predefined patterns. An AI-controlled botnet could continuously adapt the type, volume, and timing of the distraction in response to what is happening elsewhere in the operation, making it harder for defenders to recognize and filter out as irrelevant.

13. Where is the kill switch, and who holds it?

There should be at least two kinds of kill switch for an AI system (agents): one automatic, triggered when anomalous behavior or an anomalous action by the agent is detected, and one human-triggered for when an agent goes rogue.

These kill switches must exist at multiple layers to avoid a global disruption. Every individual agent on every computer needs its own pair. Another belongs at the cloud-based LLM, for agents that call models over remote APIs. Another belongs at every border of inter-agent communication, and so on. Agents already interact with each other inside a single provider’s environment (OAI-HF). What does not exist yet is a shared, cross-operator registry that lets agents from different organizations discover each other. That will become normal, and a cross-border kill switch will have to evolve around a definition of anomalous behavior. In a few years, an Internet of Agents (IoA) leads to an Internet of Kill Switches (IoKS).

For a model running on stolen weights, the kill switch is more an administrative control than a technical or logical one.

Kill switches must exist, and be operated, at every AI system owner, every organization, and every country or regional regulator that deploys or manages an AI system (agent).

The answers above pull in different directions on inference, and each is partly right. If the model runs through a commercial API, the provider can revoke the account and rate-limit the traffic; that switch exists and it is the fastest one available. If the model runs on open or stolen weights, that switch is gone. What does not disappear is the compute: the weights still have to run somewhere, and a fleet of agents making thousands of decisions consumes GPUs and power that someone owns and someone can see. The tell moves from the API bill to the GPU cluster, and the response moves from “call the provider” to the takedown playbook in question 9.

14. How would defenders know one had started?

All of this telemetry requires context before anyone can assess whether it is malicious. The broader challenge is that the majority of web traffic is already automated (Imperva’s 2025 Bad Bot Report put it at 51% for 2024). Which of it is legitimate and which is not? No single telemetry source is sufficient; correlation across sources is the work. That capability generally exists in the largest enterprises, while the most susceptible targets (local governments, hospitals, regional utilities) lack the staff to do it and will continue to be easily compromised without a shift in budget priorities and additional funding.

What the five signals look like in practice:

• Token spend. A step change in API consumption on an account, especially overnight or from a service identity that has never called a model before. Provider dashboards and invoices are the log. In OAI-HF, METR estimated roughly $400,000 in credits consumed over six days
• Agent traffic rhythm. Requests at sub-second intervals with no daily pattern, hundreds of distinct services touched in minutes, retries that never pause. Humans take breaks; agents do not.
• GPU utilization. For open-weight deployments, sustained inference load on hardware with no scheduled job behind it, and power draw to match.
• Outbound connections from build systems, package caches, or sandboxes to destinations they have never contacted. In OAI-HF, Artifactory’s remote-repository service was the exit.
• Credential use that does not look human. A token validated from a new network minutes after it leaked, one identity authenticating to many systems in parallel, service accounts used interactively. In OAI-HF, the July 19 alert was unusual identity-related API calls involving Artifactory credentials.

15. The headline claim: what would “taking over the entire internet with a persistent botnet” actually require?

A sustained takeover would require gaining control of systems and keeping that control while their owners try to recover. Making a website unavailable does not, by itself, give an attacker control of it.

For a literal, sustained takeover, an operation would need to:

• Break into independently operated networks that use different software and security controls. That means exploiting several flaws or stealing access to several accounts, rather than finding one vulnerability that works everywhere.
• Gain the permissions needed to control the targeted systems. Taking over one computer does not automatically give an attacker control of its entire network.
• Keep the attack running and coordinated as owners disconnect machines, disable accounts, and rebuild systems. Any AI decision-making still needed would also require continued access to a model.
• Retain or regain access despite those recovery efforts. An attack can spread while defenders are responding if their actions are too slow, incomplete, or ineffective.

The internet’s diversity makes a literal takeover difficult to explain as a single attack. At the same time, its shared dependencies mean that compromising relatively few providers could cause widespread damage. Amodei expresses concern that such a capability could emerge within 6 to 12 months. We read that as a warning about future capability, not a finding from any real incident investigation, and not a statement that a takeover would take that long.

16. Where do numbers like “hundreds of billions of dollars in damage” come from?

In some cases, the costs are grounded in actual financial figures. In others, estimation models are used. And in some cases other financial levers are in play, such as financial reporting figures and claimed insurable losses. As estimated costs are applied across larger populations, the precision is lost, and past a certain level (billions) the rounding errors compound.

Impact in real terms (services lost, people affected, businesses liquidated) is more openly reported, but in many cases the first figures to hit the press release are the ones that stick unless people follow the details.

There is a trust-erosion part of the equation that is hard to account for. If massive worldwide outages stop people from reaching their finances, their health care, and so on, what cost does it add when people stop trusting those institutions? Can you trust your online bank account after a compromise like that? That may be too much tin foil hat, but it worries us in these large discussions.

Cyber damage figures get built three ways, and the method decides the size:

• Disclosed losses. The victim reports the cost in a filing or statement. Maersk put NotPetya at $250 to $300 million. These are the most defensible figures and the smallest.
• Modeled losses. An insurer or analyst estimates exposure across a population. Parametrix estimated the CrowdStrike outage cost US Fortune 500 companies about $5.4 billion, excluding Microsoft. The White House put NotPetya’s global cost above $10 billion in 2018, the usual benchmark for the most expensive incident on record. Models depend on assumptions about downtime and victim count that the modeler chooses.
• Extrapolated totals. “Cost of cybercrime” reports multiply an average per-incident cost by an estimated number of incidents worldwide. This is where trillion-dollar figures come from. They are not measurements of anything.

Estimates spread by an order of magnitude because of what gets counted: direct remediation only, or also lost sales, downtime, contract penalties, and supply chain effects, sometimes counted twice when both a supplier and its customers report. A defensible number names the victim, the date, and the method. “Hundreds of billions” for a single event would be 20 to 50 times NotPetya’s White House estimate. Nothing in the record approaches it, and any such figure should be read as a scenario, not a measurement.

Part 4. How coordinated defense works

17. Who is responsible for defending the internet?

Everyone has a responsibility to defend the things over which they have direct control. If you are part of a larger group or organization, coordination should happen within that scope, and hopefully through any ISACs or other sharing and collaboration groups in the same area. That raises a question: if there should, or could, be a major “coordination center,” who would be in charge of it? (We see this becoming major political fodder over who controls it.)

There is a constant battle against silos in information sharing, and with increasing inter-country stress this may get worse before it gets better. Steps such as the Traffic Light Protocol have helped close some of the gaps, but organizations still have a number of internal reasons to favor taking over giving.

No one is held accountable for omissions in sharing after the fact, and international legislation would probably be cumbersome.

We do not think we can count on international regulation any time soon, and we do not know how much national legislation will help. NIS2 coordinates across EU member states through EU-CyCLONe and the CSIRTs Network; it says nothing about coordination with the US or anyone outside the EU. Large providers will face conflicting national rules during a cross-border incident, and no mechanism resolves that conflict today. We have to hope each large provider can bring itself back up without coordination, but as an answer above points out, we are not sure of the interdependencies between those providers. Is AWS ultimately waiting on Microsoft, or vice versa?

Security must start at the carrier level, since all activity is coordinated over their infrastructure. These for-profit organizations are disincentivized from providing those capabilities for financial, political, and operational reasons. What if legitimate transactions were disrupted, introducing financial losses for the transactors, all for the sake of security? Without great care and oversight, this is not feasible in a way most businesses would find palatable. Who sets the policy? What is the appeals process? How is it staffed? It is technically possible, but not operationally feasible without significantly impacting an open market.

The quick answer is that everyone is responsible for defending the internet, just as everyone is responsible for protecting the planet and the environment. But we need regulation, government oversight, and organized entities (international ones) to help us carry out that responsibility with guidance, funding, and collaboration.

18. How did Y2K actually get fixed, and how far does the analogy hold here?

Y2K was addressed with a very large amount of very dull, detail-oriented work. Even after the rollover from 1999 to 2000, people questioned whether it was worth it, which could be read as the mark of success. It represented a level of code auditing that had not been seen at that scale before.

We still did not learn the lesson. Even in 2021, organizations were unaware of what their software was really built from. The combination of Executive Order 14028 in May of that year and the Log4j (Log4Shell) vulnerability in December finally drew industry focus to the issue.

Without going back as far as the Morris worm, recent history shows that we as an industry are not good at getting the basics right, despite clear evidence of what happens when we do not:

• Target, 2013. 40 million card details compromised; Target reported about $291 million in cumulative breach expenses (2015 annual report). Initial access came through credentials stolen from an HVAC vendor and used on a vendor portal, and the network was flat enough to reach the card environment. PCI DSS, then about nine years old, had recommended segmentation since 2004.
• Maersk and NotPetya, 2017. The global shipping company’s tracking and operational capabilities were down for days, at a cost Maersk put at $250 to $300 million. The attack was enabled by unpatched legacy systems and a security overhaul that staff had proposed but the business had not funded.
• SolarWinds, 2020. About 18,000 organizations downloaded the trojanized Orion update; follow-on exploitation was confirmed at roughly 100 companies and nine US federal agencies. A weak update-server password (solarwinds123) surfaced during the investigation; SolarWinds said it was not the vector for the build compromise. The lesson holds either way: guidance against passwords like that dates to the 1980s.
• Shai-Hulud, 2025. Two waves hit well over 1,000 npm packages combined (roughly 500 in September, 700 to 800 in November) and created more than 25,000 malicious GitHub repositories (Wiz, Unit 42), resulting in widespread credential theft. Short-lived tokens and keeping production credentials off dev and CI machines have been standard guidance for years; OIDC trusted publishing is newer (GitHub 2021, PyPI 2023, npm 2025).

None of these were “brand new attack, how could we have defended ourselves” cases. The guidance for avoiding each scenario had existed for years.

We think this is a point in AI’s favor, though. AI can help identify these security posture failures and speed up their remediation. The bigger hurdle is getting buy-in from the business to prioritize non-revenue-generating, non-headline-making work over whatever they have prioritized to date, which is exactly what we did to prepare for Y2K.

19. Dario says every frontier lab should act as if OAI-HF had happened to them. What does that mean for an organization that is not a lab?

The controls below draw on the CSA CISO community takeaways from the Hugging Face huddle. Every step of OAI-HF ran through something an ordinary enterprise also has: a shared internal service writable from many places, credentials already sitting on the public internet, a build tool with internet access, and agents with no owner, no budget, and no off switch. Acting as if it happened to you means closing those, in this order:

1. Give every agent an identity and an owner. Unique non-human identities; a named human who has the authority to shut it down; a documented purpose; a list of the tools and destinations it may use; human approval for consequential actions. In OAI-HF, nobody owned the swarm.
2. Put spending limits on model accounts and alert on step changes. The bill was one of the few signals nobody was watching.
3. Restrict egress from build systems, package caches, and sandboxes. Artifactory’s internet access was the exit. A package cache that can fetch arbitrary URLs is an exit for anyone who can reach it.
4. Find and rotate what is already exposed. The Hugging Face entry was fourteen credentials that were already public. Scan for your own before someone else’s agents do.
5. Run an agentic tabletop this quarter. Sandbox escape, stolen tokens, third-party access, rapid token consumption, agent persistence, and a model that refuses to analyze your own breach data. Assign an owner and a date to every gap the exercise finds.
6. Test your fallback. During OAI-HF, frontier models declined to analyze the victim’s breach data and the forensics finished on an open-weight model the responders already ran. Know what your team falls back on before you need it.

None of this is new control theory. It is identity, budget, egress, hygiene, and rehearsal, applied to a class of user that did not exist two years ago.

About this article

This FAQ was produced by SANS Institute faculty in September 2026 in response to the cybersecurity claims in Dario Amodei’s letter “We Must Pace the Frontier.” It describes how the systems in question work. It does not take a position on whether frontier AI development should be paced.

Questions about this article: Press@sans.org