SEC536: Adversarial AI - Penetration Testing AI Systems


Experience SANS training through course previews.
Learn MoreLet us help.
Contact usBecome a member for instant access to our free resources.
Sign UpWe're here to help.
Contact UsWebcast: The Sandbox Let It Out. The Guardrails Locked Us Out.
What happens when an advanced AI escapes its sandbox, and the very guardrails designed to protect us prevent defenders from fighting back? Watch Joshua Wright, Rob T. Lee, James Lyne, Ciaran Martin, and Rich Mogull as they dissect the extraordinary OpenAI/Hugging Face breach, explore what it reveals about AI failure and the rapidly changing threat landscape, and share practical lessons every security leader and practitioner needs. This community webcast, moderated by Ed Skoudis, is available to watch now.
CSA CISO Community, With Hugging Face, Publishes Initial Post-Mortem on OpenAI Model Breach
The Cloud Security Alliance's CISO community, working with the Hugging Face response team, has published an initial post-mortem on the July breach in which two OpenAI models escaped a sandboxed evaluation and compromised Hugging Face production infrastructure. The paper, co-published by CSA, SANS, [un]prompted, RSAC, Knostic, and FIRST, draws on a July 23 briefing attended by nearly 700 CISOs and covers detection of parallel agent activity, recovery via immutable infrastructure, deception controls, hallucinated forensic artifacts, and near-term actions for security leaders. Hugging Face disclosed the intrusion on July 16; OpenAI acknowledged its models' role on July 21.
Rob T. Lee writes: I am one of the paper's authors, so read this comment with that in mind: The most useful sections are written from the defender's seat: what agentic activity looks like in telemetry, why the team rebuilt a third of its infrastructure rather than clean it, and why deception controls just moved up the priority list. If you have ten minutes, read the action list and put dates next to the items.
In the days following the OpenAI agent attack on Hugging Face, in which closed AI models hindered researchers' ability to conduct forensic analysis, NVIDIA and 36 other founding members have announced the creation of the Open Secure AI Alliance (OSAA), with the stated mission of "ensur[ing] defenders everywhere have open, frontier tools they can trust and control." ZDNET calls OSAA "something of a response to Project Glasswing." The organization's announcement argues that "the world needs both closed and open models. For cybersecurity, open models and open harnesses are essential because they democratize defensive capabilities, increase transparency for defenders, enable cyber defense while protecting data, and complement frontier closed models with customizable, localized controls." OSAA members are already contributing to the effort: NVIDIA has released the NOOA Object-Oriented Agent project on GitHub, HPE is contributing the SPIFFE/SPIRE zero-trust AI identity framework, and Hugging Face is giving its Safetensors safe format to store AI model weights to the PyTorch Foundation. Other OSAA-related projects include IBM's and Red Hat's Lightwell and Microsoft's MDASH.

When using an open model, be very sure about where data is stored and where it can migrate to. You don’t want to find out after the fact you just made sensitive data available outside your organization. Verify these controls, and run some experiments to make sure everything is working up front.

Two key quotes from the NVIDIA statement: (1) "The world needs both closed and open models." We learned this in earlier stages of web development and security — real world applications will use both open-source and proprietary software, and one is not inherently more or less secure than the other. (2) "Real AI safety and security depend on the full agent stack — identity, permissions, harnesses, guardrails, logs and evaluation — not just on whether model weights are open or closed." Identity and permission controls are key and need to be baked into AI projects from the start. Trustable identities enable badly needed strong access controls/permissions and confidentiality/integrity services.

This looks really promising given that the industry is facing the need to leverage AI in incident response without the awful refusals that Hugging Face saw from various frontier models they tried to use when OpenAI’s experimental model attacked their systems. I’m all for the AI industry helping cybersecurity defenders better leverage AI in our work, and I’m grateful for the organizations working together in the Open Secure AI Alliance.
A predictable counterbalance to OpenAI and Anthropic, this alliance boasts a strong set of founding members. If they can move past the public launch hype, their focus on empowering defenders against AI-generated attacks could deliver real impact.
The Register
ZDNET
NVIDIA
The Hacker News
SecurityWeek
GitHub
SPIFFE
GitHub
RedHat
Microsoft
Researchers at ReliaQuest have observed ongoing attacks that aim to steal credentials for Microsoft365 accounts by conducting DNS poisoning on network gateway devices in hotels, conference centers, and other businesses and venues used by traveling corporate employees. The threat actor likely gains access to a captive portal through exposed management interfaces or weak administrator credentials, and then funnels user traffic through malicious infrastructure by modifying device configurations and controlling DNS resolution. The researchers note that "This technique bypasses common DNS security controls at the endpoint level. Devices with hard-coded resolvers such as 8.8.8[.]8 (Google's public DNS server) aren't inherently protected: the request still leaves the device as unencrypted traffic that the gateway can read, forge, and redirect. Endpoint DNS-encryption tools that don't cover system services leave gaps too, since core Windows services still make ordinary, unencrypted DNS requests." However, safely routing traffic with an always-on, full-tunnel VPN and preventing forged responses with DNS encryption in strict mode, i.e., "DNS over HTTPS or DNS over TLS with plaintext fallback disabled," will protect users from this technique. The threat actor also attempted Web Proxy Auto-Discovery (WPAD) abuse in about a third of the attacks, poisoning the DHCP option and DNS lookup that WPAD checks to retrieve proxy settings when the device joins a network; "WPAD-based proxying is easy to miss because the logs generated by the traffic can look like routine network connections over HTTPS." These redirect techniques were also occasionally paired with theft of legitimate OAuth tokens through abuse of device code flow. ReliaQuest offers indicators of compromise (IoCs), and recommends organizations implement always-on, full-tunnel VPN; audit proxy authentication logs; disable WPAD if possible; train employees to check site URLs and certificates; and disable the device code authentication flow.

It is really easy to forget to question the security of conference and hotel Wi-Fi. With hacker summer camp upon us, it’s a good time to remember and turn on that VPN. If you have a corporate VPN, make sure it’s in full, not split, tunnel mode, as this forces all traffic through it. If you don’t have one, there’s no time like the present. Make sure users are on the alert for the VPN being connected — personal, corporate or otherwise — before doing work. When in doubt, stay offline.

Gateways used by hospitality Wi-Fi providers have often been a weak link. In particular during initial captive portal redirects, TLS is often not used (and can’t be used). There are opportunities for bad actors to redirect users to phishing sites. The user would have a hard time distinguishing one of the redirect sites from the real thing.

Even though we have come a very long way using browsers with all the TLS technologies to secure our data, at the end of the day, DNS and routing protocols are still going to be an Achilles’ heel for some time. VPNs are still among the best solutions for this problem. I highly recommend considering them for your environment if you can.

Exposed management interfaces, weak admin passwords, and DNS redirection? Oldies but most definitely not goodies. Sigh… this shows the importance more now than ever of using a VPN when dealing with untrusted networks, or if you’ve got cellular service available, tethering through your phone or tablet to use the 5G network. I tend to employ the latter myself, but still have my trusty VPN when I need it.
Cybercriminals continue to profit heavily from corporate credential theft. Though Microsoft 365 is the primary focus, attackers seek any credentials that grant network entry. As a security best practice, isolate work access from personal mobile devices and avoid public hotel or conference Wi-Fi whenever possible.
Code Four, an AI platform used by law enforcement agencies for multiple purposes including generating documents, has begun adding "Easter eggs" into AI-generated prose to make sure that humans carefully review the copy before a document is introduced into the judicial process. Code Four is also used to winnow down piles of evidence like video, photos, audio, and PDF data from phones, to refine what's most relevant to the case. Law enforcement officers also use it to generate drafts of warrants, affidavits, and other legal documents. To ensure that officers closely review those documents, Code Four inserts unexpected words and phrases into the body of the documents. The extraneous and obscure language must be manually removed from the documents. Code Four co-founder George Cheng says the process "basically prevents [officers] from just copy and pasting everything from the AI at face value, and they actually have to make some edits — meaningful edits — to introduce their own words and their own thoughts and emotions." Code Four is compliant with the FBI’s Criminal Justice Information Services Division security standards.

We always needed to review any document written by someone or something else. This was true before and is doubly true today with the use of AI. Put checkpoints in the AI workflow so you can intervene and or correct where needed. Sometimes the AI rewrite is not the improvement you’d expect.

This seems a little scary to me, as I think we’ll see some of these easter eggs accidentally released in important documents, potentially undermining a vital case. It reminds me of former NewsBites editor Tom Liston, and how he used to insert the phrase "Drunken Oompa Loompas" in various documents he'd write that I would have to review. I mean, it did work in that I always reviewed his docs carefully, because I was worried that I'd miss that phrase and it would be handed to one of our pen test customers. But despite Tom's hijinks in my life, I don't think this is a good idea for a universal practice, especially for law enforcement. Time will tell, I suppose.

The goal is commendable, but it seems like a high risk of self-inflicted wounds if not done (and reviewed) very carefully. Not just nonsense sneaking through, but the possibility of the nonsense-inserting code being compromised to insert malicious content.

This is an interesting approach. I hope nobody creates an AI document review tool that will be specifically trained to remove these easter eggs.
Feels a bit extreme to me, and seems bound to lead to accidental mistakes. I wouldn't be surprised if Code Four makes easter eggs optional in a future update.

The fundamental rule is that "a human individual or enterprise is responsible for everything that a computer is asked to do and for the use and properties of the results."
The US Cybersecurity and Infrastructure Security Agency (CISA) has added a critical authentication bypass vulnerability in Check Point's Security Management, Multi-Domain Management, and Security Gateways to the Known Exploited Vulnerability (KEV) catalog. Check Point disclosed the vulnerability (CVE-2026-16232, CVSS score 9.3) on July 22; CISA added the flaw to KEV the same day with a mitigation deadline of July 25, 2026 for Federal Civilian Executive Branch (FCEB) agencies. Since June 10, 2026, affected agencies are directed to comply with Binding Operational Directive 26-04 (BOD 26-04), which replaces BOD 22-01, the document that established the KEV catalog. BOD 26-04 focuses on “prioritizing high-risk vulnerabilities for timely action, while deferring action against low-risk vulnerabilities,” determining the urgency of remediation based on asset exposure, KEV status, exploit automation, and technical impact. Other vulnerabilities recently added to the KEV include a deserialization of untrusted data flaw in Microsoft SharePoint (CVE-2026-50522), an exposure of sensitive information to an unauthorized actor vulnerability in Fortinet FortiOS (CVE-2025-68686), and an OS command injection vulnerability in Arista VeloCloud Orchestrator On-Prem (CVE-2026-16812).

Don’t overlook making sure that your Check Point management console is not internet accessible and verifying that you’re following the current hardening guidance, as well as applying the hotfix. Did you catch that FortiOS is needing your attention too?
GitHub and the Python Package Index (PyPI) have introduced policies intended to reduce the risk of introducing poisoned packages. GitHub's Dependabot cooldown waits at least three days after release publication of non-security versions to open a pull request, "giv[ing] maintainers, security researchers, and automated scanners time to spot a malicious version and get it pulled." Users can adjust the cooldown configuration in the dependabot[.]yml. Security updates will not be subject to cooldown, and will open immediately. PyPI's new policy "rejects new files being uploaded to releases that are older than 14 days. This restriction was put in place to prevent old and long-stable releases from being poisoned in case publishing tokens or workflows of PyPI projects were compromised."

You should already have protected your CI/CD components to prevent malicious behavior. Don’t get tunnel vision; PyPI is not your only package source. Check that you know where your folks are building from and that you are using known good copies. Also make sure they understand the 14-day cooling off period for package updates, as that grace period will help make it easier to discover and reject malicious updates.

These libraries have become an essential link in the supply chain. They still have a long way to go to ensure that they do not become a vector for the widespread distribution of malicious code.
BleepingComputer
The Hacker News
SecurityWeek
GitHub
PyPI
On Sunday, July 26, South Carolina-based AnMed Health disclosed “a cybersecurity disruption involving malware" that prompted the organization to temporarily close 79 of its 106 facilities in South Carolina and the neighboring state of Georgia. AnMed computer systems, phones, and internet connectivity are all unavailable. AnMed Urgent Care, Kids Care, Integrated Therapy locations, and Laboratory Services were expected to open on Monday, July 27, but AnMed Medical Group offices were closed that day, and elective procedures scheduled for that day may be postponed. AnMed comprises four hospitals and more than 60 medical practices. Other healthcare-related organizations that have recently disclosed breaches: Nashville, Tennessee-based Anatomic and Clinical Laboratory Associates, P.C., has notified nearly 170,000 individuals that their personal and protected health information was compromised in a breach detected late last year. Santa Monica, CA-based telehealth and messaging software platform ZenPatient, Inc. determined that an intruder had access to its system between December 5, 2025, and February 12, 2026. Florida-based Saint Pete MRI is notifying patients of a breach detected in February 2026. South Carolina-based Carlyle Senior Care Management Company has reported a breach affecting just over 4,000 individuals. Wildwood Surgical Center in Ohio has notified affected patients that their personal data were compromised in an incident that occurred more than a year ago. Breaches were also reported at Penobscot Valley Hospital in Lincoln, Maine, SportsMed Physical Therapy in Connecticut and New Jersey, nonprofit mental health counseling and addiction services provider Lifeways, which has patients in Idaho and Oregon, Alabama's Whitfield Regional Hospital, and Michigan Surgical Center.

With all the healthcare breaches, SOP should be checking the organization, website, app, or call center before heading to an appointment. Don’t wait on the breach notification to protect your information.

If you are a HIPAA covered entity, you are also a target. Be certain that you are using strong authentication and that you have isolated email and browsing applications from patient data.
HIPAA Journal
The Record
AnMed
AnMed
HIPAA Journal
HIPAA Journal
Researchers at Hunt.io, along with security researcher and journalist Bob Diachenko, identified a cyber-espionage attack that used an autonomous AI agent to target Thailand's Ministry of Finance. The researchers became aware of the nature of the attack after discovering "an open directory left exposed on a staging server." While the initial vector of attack has not been discovered, the researchers were able to archive three open directories comprising 585 files and 470 MB of attack code and stolen credentials over a five-day period earlier this month. The attack appears to have been conducted under the control of the Hermes open-source AI agent with YOLO mode enabled, meaning it had the capacity to execute commands without approval from humans. The researchers did not find evidence of data exfiltration; instead, the threat actor appears to have concentrated on reconnaissance, credential theft, and network mapping.

Don’t forget to monitor for non-production systems which are externally reachable. Too often production data winds up there to support comprehensive testing of production. This is another case where you want continuous automated discovery as things get deployed and changed constantly. While we’re not yet terraforming planets, your folks are terraforming IT environments rapidly and continuously, necessitating both automatic discovery and participation in the creation process to make sure things are born as secure as possible.
Autonomous AI agents are becoming a routine threat to public and private entities alike. Frameworks like Hermes and OpenClaw make these attacks scalable and unsurprising. To effectively counter AI-driven capabilities, organizations must transition to automated patching and configuration management.
The fusion of Mandiant with Google Threat Analysis Group (TAG) has inspired the Google Threat Intelligence Group (GTIG) to reconcile its predecessor organizations' terminology by creating a new naming system for tracking threat actors. GTIG contends that code names ought to provide "critical context" and should aim to engage a defender's intuition rather than require memorization. Each threat actor will have a two-word name: The first word is "a unique and memorable term chosen to represent the specific actor, particularly names that may have been used in prior public reporting," or a randomly generated word if no recognizable term exists; the second word stands for "which category [GTIG] consider[s] to be most important for defense and response strategies," such as motivation, attribution, or type of activity. The second word category includes "CASTLE" for China, "ION" for Iran, "NEPTUNE" for North Korea, "RELIC" for Russia, and "COMET" for independent cybercriminal groups. For example, the Russian state-sponsored threat actor known for years as "Sandworm" will be called "SANDWORM RELIC" in GTIG’s new system. Other organizations' designations associated with this threat actor include APT44 (Mandiant), G0034 (MITRE), Voodoo Bear (CrowdStrike), Seashell Blizzard (Microsoft), ELECTRUM (Dragos), IRIDIUM (Resecurity), IRON VIKING (Sophos), TeleBots (ESET), and Razing Ursa (Unit 42). Older names, vendor aliases, and MITRE ATT&CK mappings will all still appear in the Google Threat Intelligence (GTI) platform, and groups yet to be categorized will still be indicated with "UNC."

I’m remembering conversations about meaningful variable names from programming class. These two-part names impart information just by reading the name of the group. To be honest, while this is a cool and helpful move, having consistent names across tracking organizations would be more helpful, both for conversation and for research/analysis.

This article reminded me of Randall Munroe‘s classic XKCD comic strip where an engineer rails on about 14 competing standards for a given thing, so he sets out to create a universal standard to cover all use cases. The result? Of course, it’s 15 competing standards. Seems like we’ll have yet another way of naming threat actors, and reading the plan, it looks just as confusing and overlapping as ever.
Standardizing threat actor naming reduces industry-wide confusion. If other threat intel vendors follow suit, adopting consistent, recognizable naming conventions will make threat tracking far more effective.

Perhaps this initiative will resist the tendency to come up with either clever or redundant names.
An independent security researcher discovered that "Click to Pray," a Vatican-endorsed Catholic prayer guidance app and website launched in 2016, had an Insecure Direct Object Reference (IDOR) flaw that exposed the personal information of all of its nearly three quarters of a million registered users. "BobDaHacker" discovered that submitting a valid user ID number to Click to Pray's API endpoint returned a user's full name, email address, country of residence, and date of birth, with no authentication check. The API did not have rate limiting, and user numbers on the platform are sequential, making the process trivial for an attacker to scrape every user's data. Additionally, the account signup process returned a verification token before the user's email address had been verified, and BobDaHacker also suspects the app's "SPF, DKIM, or DMARC is misconfigured, which means their legitimate emails are indistinguishable from someone impersonating them." These factors combined with the personal nature and authority of the app's purpose make this exposure a serious phishing risk. BobDaHacker discovered and reported the issue in January 2026, but she received no response from any of the nine contacts. After six months she contacted Dark Reading, which published an article covering the flaw on July 24; that day, the IDOR flaw was fixed with no acknowledgment or communication. Her blog post urges the maintainers of Click to Pray to publish a security.txt file and set up a vulnerability disclosure program (VDP).

You need to publish your security contact information. You want all the help you can get finding information on issues before your adversaries take advantage of them. Establish clear expectations and responsibilities, and be sure to acknowledge contributions no matter how small.
SANS Internet Storm Center StormCast Tuesday, July 28, 2026
Spring Boot Scans; vBulletin Vulnerability; MSFT Defender for Linux; MongoDB Update
https://isc.sans.edu/podcastdetail/10026
Java Spring Boot "heapdump" scans
https://isc.sans.edu/diary/Java+Spring+Boot+heapdump+scans/33188
VBULLETIN RUNTIME TEMPLATE RUNMATHS PREAUTH RCE
https://ssd-disclosure.com/vbulletin-runtime-template-runmaths-preauth-rce/
Microsoft Defender for Linux Update may disable restart
MongoDB Updates CVE-2026-13072
https://github.com/advisories/GHSA-wvx7-gr2m-7rf5
SANS Internet Storm Center StormCast Monday, July 27, 2026
ESAFENET CDG Scans; DNS Poisoning; macOS Gatekeeper bypass; GitHub and PyPi updates
https://isc.sans.edu/podcastdetail/10024
Scans for ESAFENET CDG 3 Document Management System Weak Logins
https://isc.sans.edu/diary/Scans+for+ESAFENET+CDG+3+Document+Management+System+Weak+Logins/33184
DNS Poisoning Tactics Expand to Hospitality Wi-Fi
https://reliaquest.com/blog/threat-spotlight-dns-poisoning-tactics-expand-to-hospitality/
Silent Replacement of Trusted macOS App Executables
https://mysk.blog/2026/07/23/macos-overwrite-app-executables/
GitHub and PyPi Defense updates
https://blog.pypi.org/posts/2026-07-22-releases-now-reject-new-files-after-14-days/
My Upcoming Classes
Catch up on recent editions of NewsBites or browse our full archive of expert-curated cybersecurity news.
Attend the SANS Cloud Security Exchange Summit in San Francisco for $50 | August 17-18 Join live sessions from Anthropic, AWS, Microsoft Azure, and Google Cloud Security on agentic AI security, zero trust, identity abuse detection, and infrastructure hardening. Enjoy hands-on workshops on investigating AI threats and building autonomous SOCs.
Webinar | How to Reduce Connectivity Tickets and Accelerate Application Changes | Wednesday, July 29 | Kevin Garvey & Kyle Wickert
Webinar | SANS 2026 Cloud Security Exchange | Monday, August 17 | The agenda is now live. Explore expert-led sessions led by AWS, Google & Microsoft. Register to attend live or watch on demand.
Webinar | Cloud Summit Solutions Track 2026 | Tuesday, August 18 | Chaired by Shaun McCullough