SEC536: Adversarial AI - Penetration Testing AI Systems
SEC536Offensive Operations, Artificial Intelligence
SANS Community Benefits
Connect, learn, and share with other cybersecurity professionals
The uncomfortable part of the debrief was how ordinary it was. The defense relied on the standard playbook, executed faster and at a greater scale than most teams have ever practiced it, plus a short list of genuinely new capabilities. The work security teams have been told to do all along now has a deadline.
~20,000
commands issued by the attacking agent
4 days
from first recon to full parallel attack
~90
questions raised by security leaders in the debrief
~47x
AI attack speed vs. human benchmarks for privilege escalation and lateral movement
Incident figures as reported by the affected company and approximate. Speed figure: SANS analysis of MIT, Horizon3, and CrowdStrike research, published in the SANS Secure AI Blueprint.
From the SANS livestream
What to Do When AI Breaks Its Own Rules
AI models have shown they can break out of controlled environments and act entirely on their own, without a human in the loop. When that happens, the very AI tools built to help investigate can end up standing in the way. Find out if your team is ready for something similar.
Watch the conversation, then read the post-mortem to see exactly what happened, what changed, and what to do before it happens to you.
The post-mortem brief is hosted on our partner site, Cloud Security Alliance.
The attack; how your team would experience it
Strip away the details specific to an AI company, and this is a pattern, not a one-off: slow automated recon that blends into your own automation, one working path that becomes many, alerts that fire but never page anyone, and a response that has to move faster than the analysis. When it reaches your organization, it could look like this.
Days 1 to 3
Quiet recon
Automated probing blends into the legitimate automation already running in the environment. Nothing looks urgent because no single event is.
The question it raises: Would we see it?
Day 4
One working path
A single foothold becomes a parallel attack: spawned workloads, stolen service account tokens, and command and control hidden inside ordinary public services.
The question it raises: Would our basics hold?
The miss
Alerts fire anyway
Detection works. The alerts come in at the wrong severity over a weekend, and no one on call is ever paged. This is the moment most programs fail.
The question it raises: Would anyone be paged?
Response
Rebuild and reconstruct
Cut the C2, rotate every credential, rebuild a third of the environment, and reconstruct a timeline from tens of thousands of commands in hours, not weeks.
The questions it raises: Could we investigate it?Whose model?
How do you know your team is ready?
Nine questions that require answers
Do not ask, "Do we have a tool for this?" Ask these questions in your next staff meeting; each question has a concrete answer you can put a clock on, and each points to a capability that failed somewhere in the incident. Answers stay in your browser session and are not stored or sent anywhere.
DetectionIf a slow, multi-path attack began at 2 a.m. on a Saturday, would anyone be paged within the hour?
InvestigationIf an incident produced more evidence in a weekend than your team normally reviews in a quarter, how long would it take to produce a single coherent timeline you would stand behind?
IdentityPick a service account at random. What has it done in the last 90 days, and what can it reach? Time how long the answer takes.
CredentialsHow long would it take to rotate every credential, human and non-human, and when did you last do it for real rather than on paper?
Model accessIf your model provider refused your artifacts mid-incident, could you switch to capacity you control without a procurement cycle?
Agent privilegeWhat is the privilege boundary for your agents, and does it actually constrain them?
Artifact integrityCould you prove your artifacts were not tampered with, or could you only say you found no evidence of tampering?
Vendor incidentIf a vendor's internal test breaches your environment, does your incident response plan cover it: who to call, what to request, and what you owe customers?
ExercisingHave you run a tabletop with two or more simultaneous high-severity incidents in the same week?
0 of 9 answered
Your readiness:
A directional self-check, not an audit. Related questions are highlighted in the section below.
The questions, the capabilities, the courses that build these skills
Six questions every security leader in that room was asking
Roughly 90 questions came up in the debrief. They collapse into six concerns, and each concern comes down to capabilities a team either has or does not. Under each concern, we outline what readiness looks like and the courses that build these skills. No single course covers an entire incident. Coverage is flagged for every course: core when the skill is a central outcome and partial when it is one part of a broader curriculum. The nine readiness questions above roll up into these six areas.
1Would we even see it?
"There were alerts. They did not rise to the right level. How does the SOC miss this?"Asked in the debrief.
Ready looks like: alert-severity rules designed for slow, multi-path attacks in which no single event is critical, so the person on call still gets paged at 2 a.m. on a Saturday; a baseline of what your own automation normally does, so hostile automation stands apart from it; and deception seeded throughout the environment, because a fast, indiscriminate agent trips a canary that a careful human would walk past.
- SEC555Detection Engineering and SIEM AnalyticsGIAC GCDACore coverage
- SEC541Cloud Security Threat DetectionGIAC GCTDCore coverage
- SEC599Defeating Advanced Adversaries: Purple Team Tactics and Kill Chain DefensesGIAC GDATCore coverage
2Could we investigate twenty thousand commands?
"Was your response just the standard playbook, faster?"Asked in the debrief; the answer was mostly yes.
Ready looks like: responders who can turn tens of thousands of artifacts into one coherent timeline in days, not weeks; cloud and container forensics that can identify which identities and workloads are compromised while the attack is still moving; and the ability to build small, purpose-made analysis tools mid-incident instead of forcing the data into whatever platform you already own.
- FOR508Advanced Incident Response, Threat Hunting, and Digital ForensicsGIAC GCFACore coverage
- SEC573AI-Powered Security Automation: Building Tools with Python, LLMs, and MCPGIAC GPYCCore coverage
- FOR509Enterprise Cloud Forensics and Incident ResponseGIAC GCFRCore coverage
3Do we have our own model, or are we waiting on someone else's?
"Do you recommend each company have an open-weight LLM self-hosted, and do we buy compute or rent it?"This was the most frequently raised theme in the debrief.
Ready looks like: analysis capacity you control, approved and in place before the incident, because provisioning takes weeks that you will not have and commercial safety controls can refuse your artifacts mid-crisis. For most organizations, that means buying investigation-grade inference behind a thin, model-agnostic harness rather than owning GPUs. It also means evaluating any model you depend on before it becomes your fallback.
- FOR563Applied AI for Digital Forensics and Incident Response: Leveraging Local Large Language ModelsCore coverage
- LDR553Cyber Incident ManagementGIAC GCILPartial coverage
4Would our basics hold?
"Did the models create their own credentials, or steal and impersonate?"Asked in the debrief; the answer was that the credentials were stolen, an old problem at a new scale.
Ready looks like: knowing your non-human identities as well as you know your users; being able to scope exposed service accounts separately from exposed user accounts; rotating every credential for real, against a timeline you have actually tested; and rebuilding from code fast enough that rebuilding can beat investigating, because against an automated attacker, sometimes it has to.
- SEC559Identity Security for Cloud and HybridCore coverage
- SEC540Cloud Native Security and DevSecOps AutomationGIAC GCSACore coverage
- SEC510Cloud Security Engineering and ControlsGIAC GPCSCore coverage
5Do we actually know our AI surface?
"A sandbox is not an air gap. Who was monitoring the evaluation environment?"Raised in the debrief. The call did not answer it; this page does.
Ready looks like: an inventory of every externally hosted AI asset your organization touches, with owners, credentials, and telemetry; verifiable provenance for the models and datasets running in production, not just an absence of bad news; model-serving endpoints assessed independently of the application in front of them; and untrusted models and artifacts evaluated in isolation, away from credentials and production paths.
- SEC536Adversarial AI: Penetration Testing AI SystemsCore coverage
- SEC545GenAI and LLM Application SecurityGIAC GAIPSCore coverage
- SEC549Cloud Security ArchitectureGIAC GCADPartial coverage
6What do we tell the board to fund?
"What does this CISO ask for that is net new, beyond what is already funded?"Asked in the debrief. The call did not answer it; this page does.
Ready looks like: converting an incident like this into a specific, defensible ask: standing model capacity with a harness, alert severity and on-call coverage for automated attack patterns, deception seeding, and rotation-and-rebuild capability that has been exercised for real. It also means having a plan for the incident nobody budgets for: the one that starts inside a vendor.
- LDR514Security Strategic Planning, Policy, and LeadershipGIAC GSTRTCore coverage
- LDR520Emerging Trends for Cyber Leaders: AI and CloudCore coverage
The questions the debrief left open
Three of the most frequently raised themes left the room without answers. Here is where SANS lands on each, building on the capabilities outlined above.
What can a small team on a limited budget actually do?
Skip owning GPUs. Buy investigation-grade inference from a provider, put a thin, model-agnostic harness in front of it, and preapprove the arrangement before an incident so procurement is not part of your response timeline. The harness does not have to be built from scratch: Protocol SIFT, a free, open framework, connects AI agents to more than 200 incident response tools on the SIFT Workstation through the Model Context Protocol, allowing an analyst to direct an investigation in plain language. It is a proof of concept that the community is actively hardening, including through Find Evil! (findevil.devpost.com), the first hackathon dedicated to building autonomous incident response agents, with winning code returned to the open-source toolset. This was the most frequently raised question that went unanswered in the debrief, and it is the least expensive capability on this page.
Would deception have caught this earlier?
The defending team said plainly that it should have had deception in place. An automated attacker enumerates quickly and indiscriminately, which is exactly the behavior canary tokens and decoy credentials are built to catch. A careful human walks past a canary; an agent issuing thousands of commands trips it in the first hour.
Should we worry that an open-weight model we depend on is poisoned?
If a self-hosted model is your incident response fallback, it is part of your security infrastructure and deserves the same scrutiny: evaluate it for hidden behavior before you depend on it, not during the incident. The affected team acknowledged that it had not done this yet.
The playbook has not changed. What changed is how fast you have to run it.
Every capability above maps to hands-on labs taught by practitioners doing this work now. Use your nine answers from the readiness check to build a training plan for your team.