SEC536: Adversarial AI - Penetration Testing AI Systems


Experience SANS training through course previews.
Learn MoreLet us help.
Contact usBecome a member for instant access to our free resources.
Sign UpWe're here to help.
Contact Us
Nearly every layer of how attackers operate and how organizations defend themselves leverages applied artificial intelligence. While this has been true for years with simpler incarnations of AI, ongoing developments in speed, scale, autonomy, and complex reasoning have dramatically expanded the role of AI in cybersecurity. AI is able to perform work that exceeds the limitations of human practitioners, and its direct role in cybersecurity can include discovering vulnerabilities, detecting threats, defending networks, responding to incidents, managing risk across an organization’s environment, streamlining intensive tasks with automation, and novel applications that are developing every day.
However, AI’s role is growing faster than security teams may be prepared to address. Ensuring safe, effective AI deployment requires as much rigor as confronting AI-empowered adversaries. AI is likely already a part of every organization’s environment, whether leadership and individual teams are aware of it or not, including AI use by third-party vendors, software with AI integrations, or unauthorized use of AI tools by employees (“shadow AI”).
Estimating conservatively, AI-driven attack workflows move 47 times faster than human averages, and as capabilities mature, they will likely accelerate even more, according to research cited in the SANS Secure AI Blueprint. SANS Chief AI Officer and Chief of Research Rob T. Lee notes that “MIT autonomous agent research demonstrated privilege escalation and exploit chaining in seconds to minutes compared to hours for human operators.”
The reality of AI offers both promises and demands for cybersecurity professionals. In both cases, harnessing this powerful technology requires a strategic, structured, and decisive approach to adoption.
When employed securely and efficiently, AI can analyze volumes of telemetry at a scale that no human team could process. It can identify behavioral anomalies that signature-based tools miss, correlate threat intelligence across environments in real time, and help inform analysts who make decisions under pressure.
While these capabilities can be listed like separate tools, they can only contribute to a sustainable defense when AI is intentionally integrated into a coherent organizational strategy. The SANS Secure AI Blueprint is a three-track framework designed to help organizations build that structure across their entire security program. Leveraging AI to prevent cyberattacks carries the force of the entire organization when the technology is properly protected, utilized, and governed.
The first track, Protect AI, addresses the security of AI systems themselves. Ensuring operations stay continuous and resilient requires insight into adversarial threats and collaboration between security teams, engineers, and AI/ML developers to deploy advanced defense measures.
The second track, Utilize AI, focuses on integrating AI into defensive operations while maintaining operational control, as in the specialized skills and functions mentioned above. This requires leadership from CISOs, who must invest in their SOC managers, analysts, incident responders, and threat hunters.
The third track, Govern AI, addresses leadership’s accountability. Executive suites, boards, and business leaders need to become fluent in AI and take charge of documentation, audits, and alignment with standards.
The tools themselves will be in the hands of analysts and responders, and the real-world integration of AI to prevent cyberattacks must happen at every level of the organization at once.
AI has materially changed the capabilities available to adversaries, and unique generative AI security risks can arise from configurations and controls. Security teams need a clear understanding of both.
AI lowers the technical threshold for attacks, and augments attackers’ ability to:
On-the-ground data bears this out in real attacks and research: The 2025 Verizon Data Breach Investigations Report found that over 75% of social engineering breaches showed signs of AI-assisted generation, and CrowdStrike’s 2026 Global Threat Report showed an 89% increase in AI-enabled attacks year-over-year. Horizon3’s NodeZero platform reached full privilege escalation in about 60 seconds during testing.
Defenders must also contend with risks that arise from how AI is deployed (or not deployed):
Model poisoning attacks can corrupt the training data that AI systems rely on. Mitigating this risk requires continuous validation of the model.
Prompt injection attacks use crafted prompts to manipulate large language models (LLMs), subvert guardrails, or exfiltrate data. This is the most common attack vector against LLMs, and must be addressed with layered input validation, prompt management, and intelligent monitoring.
Shadow AI is the use of AI tools not approved by the organization, which can expose sensitive data to unknown platforms. See the FAQ below for a SANS blog on how to address this challenge.
Data leakage can occur due to misconfigurations and errors, potentially exposing the proprietary data that powers the model. Controlling for this requires auditable end-to-end logging, strict permission models, and comprehensive encryption.
SEC598: AI and Security Automation for Red, Blue, and Purple Teams is designed to give security practitioners hands-on experience with these attack techniques, helping them ground defensive architecture in an accurate understanding of how the threats actually work.
Yes, and doing so is increasingly a practical necessity. Most SOCs operate with between two and ten full-time analysts, a number that has held steady since SANS began tracking it in 2017, while the scope of required coverage has expanded to include cloud environments, remote endpoints, SaaS platforms, and other resources. AI automation can be a means to handle the disproportionate scale.
The SANS Secure AI Blueprint elaborates that “AI offers a practical path forward by optimizing parts of the job that slow analysts down: the disorganization, repetitive steps, and cognitive overhead.” When paired with capabilities like Model Context Protocol, language models can “integrate telemetry, threat intelligence, asset metadata, and user history into a single view, tailoring it to each unique situation the analyst faces. This gives analysts enriched, case-specific summaries instead of raw events.”
In addition to SEC598, more SANS courses build the specific skills required to put this automation into practice.
SEC595: Applied Data Science and AI/Machine Learning for Cybersecurity Professionals readies teams to build AI-driven detections.
SEC503: Network Monitoring and Threat Detection In-Depth trains teams to automate ML-based anomaly detection on NetFlow and packet data.
SEC511: Cybersecurity Engineering: Advanced Threat Detection and Monitoring covers engineering detection pipelines that defend generative AI and LLM applications and apply ML-enabled continuous monitoring.
The outcome for security teams must be to operate at a speed and scale that matches an AI-enabled adversary, while preserving the human judgment that will always be indispensable for strategic decisions.
Visibility is the first step of governance and strategic deployment: Leaders should examine how people across an organization are using AI before considering what should be done about it.
Despite the perception of AI as a strategic imperative, many of the urgent directives for deployment come from “business-line leaders with minimal understanding of how to implement, scale, and govern AI effectively,” according to the SANS Secure AI Blueprint. Two-thirds of respondents in a Deloitte survey titled Governance of AI: A Critical Imperative for Today’s Boards, reported “limited to no knowledge or experience” with AI, and less than a third of enterprise organizations have established a comprehensive AI governance framework.
A robust governance, risk, and compliance (GRC) program for AI also begins with visibility, and must cover third-party AI tools, vendor-supplied models, and unauthorized deployments operating outside standard workflows.
Three components are foundational to a well-designed AI GRC framework:
An AI Bill of Materials, or AIBOM, modeled on the software bill of materials concept, is a live inventory of every model, dataset, API, and configuration in an organization’s AI environment. It gives security teams and auditors a clear view into where models operate, what data they access, and how they behave.
Model registries track lineage, versioning, performance, and changes to models in production. Without this line between experimentation and production, changes can occur without review or documentation.
Executive and cross-functional AI committees establish the systems that govern deployment, escalation, and accountability. High-performing organizations are forming these bodies, often chaired by a Chief AI Officer, Chief Risk Officer, or designated AI lead, drawing from legal, compliance, security, IT, and HR.
In March 2026, the SANS Institute published the Critical AI Security Guidelines v1.4, aligned with the OWASP AI Exchange, to provide actionable guidance across eight control categories.
1. Conventional security controls are known security fundamentals applied to AI assets.
2. Data minimization and obfuscation addresses the elevated risk of data leakage specific to AI systems.
3. AI supply chain management requires scrutiny of where models come from and how they are hosted.
4. Data and model engineering controls cover the practices that AI and ML teams apply during development.
5. Limiting model behavior is an aspect of zero model trust.
6. Model I/O handling filters the inputs and outputs to protect against manipulation and unauthorized interactions.
7. Monitoring must be continuous.
8. Governance, risk, and compliance brings the technical controls together under organizational accountability.
For a mapping of SANS training to these frameworks, visit the SANS frameworks and directives page.
As the SANS Secure AI Blueprint states: “The global race to adopt AI is well underway, but the readiness to secure it is still trailing behind. Most organizations are deploying AI faster than they can govern, integrate, or defend it, leaving critical gaps in oversight, infrastructure, and trust. What’s needed now is a stronger commitment to secure AI adoption backed by control, transparency, and accountability from the ground up.”
The path forward requires that we treat AI security as foundational. This creates a mandate for decisive ownership of how the technology is protected, utilized, and governed, involving every level of the organization. Security teams that build that foundation now, through structured training, disciplined controls, and board-level governance, will be positioned to handle the continual evolution of both threats and defensive capabilities.
How do AI-powered attacks differ from traditional cyberattacks?
AI lets attackers move faster, personalize attacks at scale, and automate what used to require significant human effort. Traditional security controls weren’t built for this tempo. Modern defenses need to match it.
Hear Ryan Chapman, Mari DeGrazia, and Raymond DePalma discuss weaponized AI: Stay Ahead of Ransomware: The AI Arms Race – When Both Sides Have Copilots
What is shadow AI, and why should organizations care?
When organizations slow-walk AI adoption, employees find their own tools. That means sensitive data flowing to unvetted platforms with no visibility or control. The fix starts with a clear acceptable use policy and knowing what tools are actually in use.
Listen to Rob T. Lee’s interview with Wade Foster on the balance organizations must strike when managing AI use: Wade Foster, Zapier CEO: Why Saying ‘No’ Makes Shadow AI Worse
What is prompt injection and why is it dangerous?
Prompt injection tricks an AI model into ignoring its instructions and doing something it shouldn’t. It’s the most common attack against large language models, and it can’t be solved at the model layer alone. Defense requires input/output controls, guardrails, and monitoring working together.
SANS’s updated course on AI-powered security automation teaches how prompt injection attacks work and how to implement security controls against them: Automating Security with AI: What’s New in SEC573
What is the role of AI in cybersecurity?
AI gives security teams the ability to detect threats faster, automate response, and operate at a scale that manual processes can’t achieve. It also introduces new risks that need to be managed. The goal is to make AI a genuine advantage for defenders, not a liability.
John Hubbard writes about the state of the SOC in the age of AI, and James Tarala proposes approaching GRC as a unified lifecycle, including AI considerations:
The Augmented Analyst: How AI is Reshaping Security Operations in 2026
Beyond Risk Assessments: Building and Running a Cybersecurity GRC Program


With a 15-year career that has spanned telecommunications, geospatial analysis, machine vision, data privacy, and more, Jason Hodgert dedicated his career to making technologies logical, accessible, and understandable to mass audiences.
Read more about Jason Hodgert