SEC536: Adversarial AI - Penetration Testing AI Systems


Experience SANS training through course previews.
Learn MoreLet us help.
Contact usBecome a member for instant access to our free resources.
Sign UpWe're here to help.
Contact Us
This month's SANS Threat Analysis Rundown took on a story that moved faster than almost anything else in cybersecurity this year: the June 2026 export-control episode that pulled Anthropic's two most capable AI models offline worldwide overnight, then reversed course less than three weeks later. To work through what that actually means for defenders once the headlines fade, I was joined by Thomas Roccia (@fr0gger_), creator of the NOVA prompt-hunting framework and the Unprotect Project, longtime Microsoft threat researcher, author of Visual Threat Intelligence, and founder of Security Break, a company focused exclusively on AI threat intelligence.
STAR stays practitioner-first: no vendor pitches, no recycled headlines, just a working-level conversation about what's happening and what defenders can do about it. Thomas and I have also co-authored a SANS blog, Mythos: Forget the Model, Follow the Workflow, publishing shortly after this episode, and this conversation gave a first look at its central argument. That thesis is worth sitting with: Mythos is real and impressive, but almost none of what makes it scary is new. You could already do this, and that is the part that should change how you defend. You can catch the full replay of this episode on the SANS DFIR YouTube channel.
We opened on the news itself. In June 2026, days after Anthropic shipped Fable 5 broadly and Mythos 5 to vetted partners, a US export-control directive barred foreign nationals from accessing both models. Because nobody can verify nationality on every request in real time, the practical result was blunt: both models went dark for the entire planet overnight while everything else kept running. Less than three weeks later the controls came off, with Fable returning globally and Mythos restored to a vetted set of US organizations.
We deliberately steered away from relitigating the politics of the directive and stayed on the practitioner question. Thomas's honest read from a defender's seat was frustration: a new model is a new advancement, and losing access to it stings, even if the national-security instinct behind the decision is understandable. But we landed in the same place, and it's the same place as the 76 senior security leaders who signed the open letter at freefable.org asking for the controls to be lifted: the restrictions likely hurt defenders more than attackers.
The logic is straightforward. Sophisticated adversaries had already engineered multiple models into capable systems long before Fable or Mythos existed. They didn't wait for a frontier model, and they don't apply for export licenses. The people actually slowed down by a control like this are the defenders who were using a supervised model as part of their workflows, triage, and research, and who may be less equipped to stitch together their own multi-model harness. Katie Moussouris, the one outside expert who says she actually read the research behind the directive, made a related point that's worth carrying into any policy room: what the model did was the find, fix, and test loop defenders run every day, and you can't export-control your way to cyber resilience.
The durable takeaway isn't about one directive. It's that your most capable model is now something a government can switch off with a letter. Not an outage, not a deprecation, a policy decision executed overnight. That reframes model access as an operational risk, which is a thread we picked up again at the end.
Both Thomas and I spent weeks hands-on with Fable, and we hit the same wall. Fable is a genuinely strong model, and its headline improvement is that it needs far less prompt engineering than Opus 4.8. You ask, and it does the task well without much iteration. My shorthand for it is the "easy button": it's great for someone who isn't a prompt-engineering specialist and isn't working on anything security-related.
The catch is the guardrail behavior. Anything with even a hint of cybersecurity triggers a downgrade back to Opus 4.8. Thomas described it as a cement wall, and in our testing, that was accurate: even tiny security-adjacent questions, and sometimes questions that weren't security-related at all, bounced the session back to Opus. These guardrails are likely tied to the emerging industry effort (Anthropic working with Amazon, Microsoft, and Google) to build a consensus framework for scoring the severity of AI jailbreaks along dimensions like capability gain, breadth, ease of weaponization, and discoverability.
Thomas walked through his workaround, which is instructive precisely because of how fragile it is:
The practitioner lesson underneath the workaround is the real point: guardrails applied at the model layer are brittle as a control. They can be worked around with enough patience, which is part of why the model was restricted in the first place, and they punish plenty of legitimate, non-malicious work along the way. A control you can defeat by reorganizing your folders is not a control you should lean on, in either direction.
If there's one idea from both the episode and the blog worth internalizing, it's the AI harness. Thomas frames it as the evolution of what we called context engineering in 2025: the whole architecture around the model, not the model itself.
The distinction he draws is between the model (the brain, where the generated knowledge lives) and the agentic system built around it. On its own, an LLM is limited. It can't fingerprint open ports, disassemble a malware sample, or run a scanner. The harness is everything that gives the brain hands: the prompts, the tools and MCP servers, the skills, the memory, the API connections to internal and external resources, and the workflow that ties them together, often with guardrails and regression testing so the agent can write code, test it live, take the feedback, and iterate. Claude Code, Codex, OpenClaw, and provider SDKs are all examples of harnesses.
Here's why it matters: a well-built harness with the right knowledge and workflow can already do serious work, including vulnerability research, with the models we already had. Mythos, connected to the right harness, can chain exploits and reason across a complex codebase, but this was already possible before Mythos. What changed is how little engineering it now takes. That is the whole story, and it's the one sentence to carry out of both the episode and the blog.
The blog assembles the receipt trail behind that claim, and it predates anyone whispering the word "Mythos." A few of the pieces we pointed to:
The point isn't that the model doesn't matter. It's that the next big disruption probably comes from a harness, not a model, and defenders who fixate on the model name are watching the wrong variable.
If the harness is what does the work, then export-controlling one supervised US model doesn't make the capability vanish. It relocates: to other frontier models, and to open-weight models with no brakes at all.
Thomas made the point directly with GLM and similar open-weight Chinese models. They ship without Western cybersecurity guardrails, and once the weights are public, nobody is putting them back in the box. They can be fine-tuned for any purpose, which makes autonomous vulnerability research plausible today, not hypothetically. The easier the capability is to access, the more we should expect attackers to fold it into their toolboxes.
He added a wrinkle worth flagging for anyone building defensive pipelines on these models. They aren't unguarded; they carry guardrails aligned to Chinese government sensitivities instead. That's not just a content-policy quirk, it's a potential attack vector. If you run malware analysis or another defensive workflow on a model with China-specific guardrails, an attacker could embed prompts designed to trip exactly those guardrails and break your analysis pipeline. The guardrail becomes something an adversary can weaponize against your own tooling.
The heart of Thomas's current work, and of Security Break, is a problem most teams haven't reckoned with yet: when you run an AI agent, you don't actually know what it's doing in the background. Agents operate at lightning speed, calling tools, running commands, and connecting to internal and external resources autonomously. You get the result, not the trace.
This is the thread that runs from NOVA to what Thomas is building now. A quick tour of that evolution, because it explains why prompts alone weren't enough:
Why does tracing matter so much? Because supply chain risk lives here. An agent can install a malicious npm or PyPI package in the course of doing its job, and if you can't see what it's doing, you can't catch it. Thomas frames observability as the first requirement, not just for troubleshooting but specifically for security: once you can see what the agent is doing, you can fire detections, alert, and run incident response.
The critical nuance is that detection here can't be hard-coded rules on single actions. A single tool call or bash command might look suspicious in isolation and be perfectly benign in context, or the reverse. Effective detection requires the full context of a session, reasoning across the whole trace. The analogy we kept coming back to is EDR: an EDR doesn't just flag an event, it gives you the chain of events that led to it. Detecting threats in an agent session is similar, but harder, because you have to hold the entire session's context to know whether a given step is actually malicious. OpenTelemetry is genuinely useful for observability and troubleshooting, but it's not sufficient for security detection, because much of what you need for that isn't captured there.
We spent time on the rule the blog argues every team that botches AI integration breaks: AI suggests, the human decides, automation performs.
Thomas's version of this is that we're still early, hallucination and unauthorized actions are still real risks, and so humans stay in the loop for high-sensitivity tasks like security triage and research, at least for now. AI today is an excellent assistant that helps teams move faster, not a replacement for the analyst.
His forward-looking framing was the one that landed hardest. For years the conversation has been about AI replacing SOC analysts. But AI is a force multiplier for attackers too. As adversaries push out more AI-generated attacks, defenders need more people, not fewer, to monitor and orchestrate the agents responding to them. The plausible near future is security teams managing multiple agents in production, orchestrating them to analyze attacks carried out by adversary agents. Call it agent vs. agent. It sounds like science fiction, but the point isn't to get rid of the people running security, it's to train them to operate at scale against technology that's moving fast. As I put it on the stream, it's time to level up.
The SANS RSAC 2026 recap is a good companion here: Heather Barnhart's frameworks map AI integration onto NIST CSF 2.0 and the forensic lifecycle, letting AI assist Detect and Prepare while keeping Recover, lessons learned, legal and evidentiary conclusions, and attribution human-only. The trap is the autonomy gap: finished, confident output arriving faster than your review process was built for. Design the checkpoint on purpose, before you learn the hard way that you needed one.
I asked Thomas where the identity landscape goes as organizations spin up fleets of agents, because it feels like it's going to be a mess. His answer confirmed it, and the failure mode he named first was the obvious one: agents accumulating too many permissions.
Agent identity is genuinely early. The most concrete work he pointed to is Microsoft's Entra Agent ID, released only a couple of months ago and Azure-focused for now, alongside efforts to extend open workload-identity standards like SPIFFE (the vendor-neutral, CNCF-hosted Secure Production Identity Framework for Everyone) to agents. His expectation is that these efforts eventually converge toward a common standard.
The reason agent identity is harder than user identity is that an agent is not a user. It acts at high speed and can do an enormous amount in a very short window, which makes monitoring and governing it far messier than tracking a human's access rights. My bet, and Thomas agreed, is that this evolves into something like an insider-risk or insider-threat module, but built specifically for the agents operating inside your environment.
June's export ban proved that a team's most capable model can become a revocable dependency overnight, so we closed the strategic thread on treating model access the way you already treat asset inventory.
The clean approach Thomas recommends is centralization: route AI access through managed services like Azure AI Foundry or Google AI Studio so the organization controls which models are available and maintains visibility into how they're used. That's the simplest way to know what you have.
The harder half is shadow AI: employees using personal API keys and unauthorized models on their own machines. You can't inventory that through the managed-service front door. It takes dedicated tooling to do internal reconnaissance and build a posture picture of what people are actually running. Once you can answer that question, you can give users sanctioned managed-AI options that meet their needs, instead of driving the behavior further underground.
This ties directly to the blog's "revocable dependency" guidance: own a model-inventory and model-redundancy function, because if you wired a single frontier model into your workflows, you're one outage, deprecation, or directive away from it going dark. June proved the point.
One thread that came up unplanned, in the best way: disinformation. Thomas noted it's a category inside the MITRE ATLAS matrix for AI threats, and the reason it's on his radar is that the early OpenAI threat reports flagged nation-state actors using models to generate disinformation, including an example of a Russian influence campaign using GPT-style models to generate fake images around the Paris Olympic Games.
I raised the DISARM Framework as a potential complement, an ATT&CK-style framework focused specifically on disinformation techniques and countermeasures, and we talked through whether NOVA's IOPC data and a framework like DISARM could feed each other to support both detection and response. It's an idea worth developing, and a good example of how AI threat intelligence keeps colliding with existing CTI tradecraft rather than replacing it.
We closed on the practical question every STAR episode comes back to: what do you actually do Monday morning, especially on a limited budget?
Thomas's answer starts with two questions, because most organizations are still early in their AI journey:
The one thing he stressed is not to answer the first question in a silo. Whether you need AI, and how much to restrict it, depends heavily on your industry, your data sensitivity, whether you're critical infrastructure, and how much your developers depend on it. Blocking AI outright without talking to stakeholders across the organization is a good way to either break your engineers' workflows or push all the usage into the shadows. Talk to the relevant stakeholders, then decide.
For the rest of 2026, the blog flags three things to watch: whether Mythos expands past the vetted US organizations, whether the jailbreak severity framework becomes a triage standard, and how fast open-weight models close the capability gap. And the message underneath all of it, from both the episode and the blog: if you're using AI, so are threat actors, and the asymmetry is cultural. Teams that won't put AI on defense can't keep pace with the ones putting it on offense. Forget the model. Follow the workflow.
The Episode
Thomas Roccia, Security Break, and NOVA
The Mythos and Fable Export-Control Story
Harnesses, Autonomous Research, and the Receipt Trail
Defender Programs, Frameworks, and Identity
Frameworks and Reference Material
If you want to keep building on what we covered this month:
Next month on STAR, we'll be back with another guest and another working-level conversation on what's actually moving in the threat landscape. And keep an eye on the SANS blog and both of our feeds for Mythos: Forget the Model, Follow the Workflow, the deeper written companion to this episode that Thomas and I co-authored, publishing shortly. Catch the replay of this episode and register for upcoming livestreams at the SANS Threat Analysis Rundown page.
Thanks to everyone who joined live or is catching the replay, and a special thanks to Thomas Roccia for cutting through the hype and showing defenders where the real work is. See you next time.


Sean O’Connor is a cybersecurity professional with over 15 years of experience in intelligence, digital forensics, and threat analysis across both the private and public sectors.
Read more about Sean O'Connor