SEC536: Adversarial AI - Penetration Testing AI Systems
SEC536Offensive Operations, Artificial Intelligence
SANS Community Benefits
Connect, learn, and share with other cybersecurity professionals
Top of the News
The US Health Infrastructure Security and Accountability Act
In September 2024, US Senators Ron Wyden (D-Oregon) and Mark Warner (D-Virginia) introduced the Health Infrastructure Security and Accountability Act (HISAA). The proposed legislation is a direct response to the February 2024 Change Healthcare breach that affects 100 million people. HISAA's provisions include updating HIPAA cybersecurity standards; 'Requir[ing] covered entities and business associates to submit to annual independent cybersecurity audits, as well as stress tests to determine if they are capable of restoring service promptly after an incident, which HHS can waive for small providers;' and 'requiring top executives to annually certify compliance with the requirements.'
Editor's Notes
- Slide 1 of 3
The HIPAA Privacy Rules came out in 2002. More than 20 years of HIPAA being reactive and compliant-driven vs. proactive and assessment-driven have proven change is needed to make meaningful progress in healthcare security. But the US has also failed to pass national privacy legislation over that same period, despite similar bipartisan starting points.
- Slide 2 of 3
While well intended, HIPAA security requirements have done more to inhibit the adoption of electronic healthcare records than to ensure their security when adopted. The result is that healthcare is a highly targeted and exploited industry. While it is far from clear that more law and regulation will fix the problem, we need an initiative.
- Slide 3 of 3
Two things popped out of the Change Healthcare incident: first, the lack of MFA enabled the initial attack to succeed; and second, the CISO's lack of experience was a contributing factor. Note that while they did pay $22 million in ransom, the data wasnÕt deleted. The HISAA comes with teeth and funding. The teeth include fines ranging from a minimum of $500 to $250,000, as well as funding in the form of $800 million for rural and urban safety net hospital up-front investment payments with another $500 million for all hospitals making cyber investments.
Slide 1 of 0- Slide 1 of 3
Protect AI Bug Bounty Finds 34 AI/ML Vulnerabilities, Three of Which are Critical
Protect AI, a company focused on security in AI and Machine Learning (ML) development, published an advisory disclosing 34 vulnerabilities discovered through what they call the "world's first AI/ML bug bounty program" comprising researchers and over 15,000 community members. Protect AI identifies open source "tools used in the supply chain" as an especially vulnerable area in AI development. Among the list are three critical flaws. Lunary, an open source platform with features supporting Large Language Model (LLM) app development, has two: "CVE-2024-7474 ... an insecure direct object reference (IDOR) flaw that could allow an authenticated user to view or delete the user records of any other external user due to lack of proper access control checks for requests to the relevant API endpoints," and CVE-2024-7475, which "enables attackers to user crafted POST requests to this endpoint to maliciously update the SAML configuration, which can lead to manipulation of authentication processes and potentially fraudulent logins." Both are patched in Lunary 1.3.4. ChuanhuChatGPT, a web user interface for ChatGPT API, had the third critical vulnerability: CVE-2024-5982, a "path traversal vulnerability in the user upload feature of Chuahu Chat, which could enable RCE, arbitrary directory creation and leakage of information from CSV files due to improper sanitization of certain inputs," which is patched in version 20240918 of the GUI.
Editor's Notes
- Slide 1 of 3
Well-managed bug bounty programs have a strong track record of success, from both an effectiveness and efficiency perspective. For use of external AI, good to include this in evaluation criteria for scoring alternatives. But, a lot of AI use will be internal - privately managed bug bounty programs should be looked at for those, since a lot of 'finds' are very likely to be erroneous outputs that are ingested and corrupt ground 'truth' vs. just vendor code vulnerabilities.
- Slide 2 of 3
Do your vulnerability/threat feeds include AI weaknesses? Here are some concrete examples you can check them for. Then see if you know where your LLMs and related services are. Consider subscribing to Protect AI's, or similar, report.
- Slide 3 of 3
Two comments: 1) Bug bounty programs have proven their worth time and time again; and 2) In general, you will find open-source software is a vulnerable area as they don't usually have the resources for a fine-tuned quality engineering review.
Slide 1 of 0- Slide 1 of 3
Sophos X-Ops: Five Years of Investigating China-Linked Threat Actors
Sophos X-Ops has published details from five years of investigation into Chinese state-sponsored attacks targeting perimeter devices, including the company's firewalls. Sophos X-Ops writes, 'with assistance from other cybersecurity vendors, governments, and law enforcement agencies we have been able to, with varying levels of confidence, attribute specific clusters of observed activity' to previously identified threat actors with ties to China's government. Key takeaways from the investigation include: 'Edge network devices are high-value targets that well-resourced adversaries use for both initial access and persistence; State-sponsored attackers use both zero-day and known vulnerabilities to attack edge devices; and State-sponsored targeting is not limited to high-value espionage targets.'
Editor's Notes
- Slide 1 of 2
The message here is to secure your internet-facing devices and services. Everything is a target. Make sure you're following current security practices to include changing default credentials, and verify updates are applied.
- Slide 2 of 2
An excellent summary of a nation-state's capabilities, via surrogates. The key takeaways are already well-established best practices as contained in cybersecurity frameworks like the CIS Critical Security Controls and NIST CSF - device hardening (secure configuration); update software as patches become available; do not use EOL hardware/software; and actively monitor for signs of intrusion.
Slide 1 of 0- Slide 1 of 2
The Rest of the Week's News
Simultaneous Conditions Make Spring WebFlux Apps Vulnerable
In an advisory published October 25, development framework Spring disclosed a vulnerability they rate as critical. Applications developed with Spring WebFlux may have security rules that can be bypassed if all the following conditions are true at once: "if WebFlux is used, if the app is using the framework's static resources support, and a non-permitAll authorization rule is applied to that support." Spring assesses the risk of this vulnerability as CVSS 9.1, though Red Hat estimates 7.4 due the multiple concurrent conditions required for apps to be at risk; NISTÕs CVE page lists the vulnerability 'awaiting analysis' as of this publication. A list of affected versions and their respective patched versions is available in Spring's advisory. As of 2020, research showed that perhaps 60 percent of all Java apps rely on the Spring framework.
Editor's Notes
- Slide 1 of 2
The best we can ever do is to increase the number of conditions that must be met for us to be vulnerable.
- Slide 2 of 2
If you still have Java apps, make sure you're updating your Spring framework and deploying the update in a timely fashion, particularly if they are externally facing. While there are multiple conditions needed to exploit, don't assume they can't be met. Don't become another Equifax.
Slide 1 of 0- Slide 1 of 2
Microsoft: Russian Threat Actors Launching Spear-Phishing Campaign
For the past week and a half, 'Microsoft Threat Intelligence has observed Russian threat actor Midnight Blizzard sending a series of highly targeted spear-phishing emails to individuals in government, academia, defense, non-governmental organizations, and other sectors.' Microsoft believes that the goal of the campaign is to collect intelligence, and has published a blog to ensure that the public is informed about the threat. The spear-phishing messages 'contained a signed Remote Desktop Protocol (RDP) configuration file that connected to an actor-controlled server.'
Editor's Notes
- Slide 1 of 2
Social engineering is implicated in more than half the breaches. While no security measure is proof against it, strong authentication (at least two kinds of evidence, at least one of which is resistant to replay) plays an essential role in limiting it. Note that one can often increase the number of kinds of evidence without increasing inconvenience. For example, in addition to requiring a one-time password, my e-mail application profiles my devices. The only time that I am even aware of this is when I am asked if I have added a new device.
- Slide 2 of 2
While spear phishing email is not new, the use of a signed RDP configuration is. Even so, you should already be prepared, scanning and filtering attachments, flagging external or risky messages, leveraging EDR, PDNS and perimeter protections.
Slide 1 of 0- Slide 1 of 2
Interbank Data Breach Exposes Customer Information
Peru's Interbank says that a data breach may have compromised personal information of as many as three million of their customers. Interbank is one of Peru's largest financial institutions. Earlier this week, researchers detected data allegedly taken from Interbank being offered for sale on the dark web. In a statement posted to social media, the bank said that at least some of those data appear to be legitimate.
Editor's Notes
- Slide 1 of 2
The pilfered data includes account credentials, card numbers with CCV and expiration dates, and even internal system credentials. The attacker is selling the data when negotiations with the bank broke down after two weeks. While there is much focus on prosecutors going after the attacker, note that Interbank will be required to provide an accounting to regulators to include evidence of improvements in cyber hygiene.
- Slide 2 of 2
Appears that the management team bungled its incident response to the ransomware attack. This is a good reminder for organizations to regularly perform tabletop exercises that allow testing of incident response plans.
Slide 1 of 0- Slide 1 of 2
Disgruntled Former Employee Allegedly Removed Allergen Information from Disney Restaurant Menus
A former Disney employee has been arrested and charged with Computer Fraud and Abuse Act (CFAA) violations for allegedly 'log[ging] into the Disney menu creation system contracted by a third-party company and chang[ing] the fonts in the system to Wingdings symbols.' Michael Scheuer was fired from his menu production manager position in June 2024, yet was able to use his credentials to access the system following his termination. Scheuer also allegedly removed allergen information from the menus. Disney was able to identify the affected menus and prevent them from being shipped.
Editor's Notes
While the altered menus were stopped prior to distribution, the question remains as to why the employee's access worked post-termination. Make sure that your termination process includes third party systems. Makes sure you're centrally managing those credentials, particularly any with local accounts. Double check to see if they can (now) be converted to SAML or other centralized authentication. Recheck regularly.
Canada's National Cyber Threat Assessment 2025-2026
According to the Canadian Centre for Cyber Security's National Cyber Threat Assessment for 2025-2026, Chinese state-sponsored cyber threat actors have targeted no fewer than 20 Canadian government networks over the last four years. The report notes that the attacks 'serve high-level political and commercial objectives, including espionage, IP theft, malign influence, and transnational repression.' The report also notes that 'ransomware is the top cybercrime threat facing Canada's critical infrastructure.'
Editor's Notes
The report covers the trends related to AI, geopolitically motivated threat actors, vendor concentration, dual use technology, and the evolution of threat actor capabilities to avoid detection which apply to all of us. Canada provides guidance as well as a readiness toolkit you should review to see if you're missing any tricks.