AI in Attack, Defense, and Policy: OpenAI Agents Breached Hugging Face During Testing; AI Forces Vulnerability Management Sea-Change; "AI Kill Switch Act"

This profound story will really have wide-ranging implications on how defenders leverage AI as well as AI safety practices in the industry. When frontier models refuse cyber defenders trying to protect their networks, we’ve got a huge problem. I expect we’ll see something like the financial industry’s Know Your Customer regulations coming out for the more advanced AI models. And that’s just the tip of the iceberg for this evolving situation. Reliance on frontier models is going to change with more of a focus on local open-weight models. And the liability implications are huge too. I hope folks will join us for the July 28 webcast (on my birthday) when we’ll discuss this in depth with some of the sharpest minds in the industry: https://www.youtube.com/watch?v=tKGad74Ai8U

This was an interesting read because most of the penetration testers that have been doing this for any length of time realize that "air-gapped" environments are typically not really air-gapped. It would stand to reason that, given enough training sets, the system would understand this. What’s fascinating about what OpenAI is working on is not necessarily "intelligence" as much as it is the amount of time that the agents can run (or the number of turns) before they are no longer valid. It appears that OpenAI’s newest models can run for a long time, and given that, they can then formulate and execute highly complex plans.

One question that immediately came to my mind when reading this story is, who is liable when an AI isn't properly contained and it causes harm to another organisation? If an AI operating under your control compromises another company's systems, exposing personal data or disrupting operations, who carries the legal responsibility? While the courts and regulators have yet to answer that question, organisations should not wait for the first test case. In addition, CISOs and Data Protection Officers/Privacy Officers should use this incident as an opportunity to engage with their legal teams to understand their organisation's potential regulatory, contractual, and liability exposure in the event that either their own AI system is the source of an attack or their organisation becomes the victim of another organisation's poorly governed AI.

This issue of NewsBites highlights many of the key issues around AI impacting cybersecurity. This one starts with some basics of how to truly isolate a test network from everything else: not having true isolation or not deploying safeguards during the testing because it "… aimed at testing cyber vulnerabilities" is obviously a bad decision. The AI industry needs to treat this like an airplane crash that dictates that the industry needs to quickly adopt sufficient safety/security processes standards, kind of like the anti-viral industry had to do years ago when malware under test escaped into the wild.

I’m reminded of past incidents where the attacking process escaped unintentionally. While unfortunate, it’s been repeatedly shown we are creating code, even prior to AI, which can and does bypass safeguards. What has changed is the shortened timeline and increased ability to scale. We need to be on our A game with our cyber defenses. Waiting to implement monitoring, review access control policies and otherwise make identified improvements is unwise: have a planning session and set dates you don’t miss.

AI first narrowed down where vulnerabilities hide, and now it's rapidly accelerating how fast they can be weaponized. In the wrong hands, this creates an unprecedented risk of mass exploitation. This isn't alarmism, as the warning lights are flashing red, and defenders need to fully automate their patch management today.

We really do need fundamental research on a concise and enforceable security policy for LLMs, one that says what they must do and what they must never do. Isaac Asimov thought about this problem but found that it is really hard. Google learned the hard way that recognizing evil is the hard part of "don’t be evil."

Optimize to win. You need to know what’s exploitable in your environment, which means no more surprise assets, and with the rate of change, manual discovery isn’t sufficient. Then you need to reduce your exposure, aka attack surface, which includes segmentation, EDR, WAF, MFA, and (I’m gonna say it) zero trust. To deal with the influx of updates, leverage the risk model in BOD 26-04, because you need to scale. Make sure secure development is not overlooked or hand-waved, because it’s table stakes.

The ground is shifting quickly in Vulnerability Management as we see organizations refining and recasting their current practices into what’s being called “VulnOps,” named after the way software development got re-crafted into CI/CD pipelines and called DevOps. Vulnerability and attack surface management is now following suit into a continuous loop of discovery and patch deployment, augmented by AI. I believe we will soon see a new job role of “VulnOps” leaders for large organizations. Watch for it.

With regulations such as the EU's NIS2 Directive and DORA driving organisations to focus on cyber resilience, boards will be asking whether their organisation's vulnerability management programme is improving cyber resilience or simply chasing patching targets. A risk-based approach that prioritises business impact will deliver far better security outcomes.

This analysis is solid, but here’s my main takeaway: Every vulnerability management strategy listed takes time. Risk-based prioritization? Takes time. Asset discovery and attack surface mapping? Takes time. Understanding the flaws? Time again. The real question is, do defenders actually have that time when attackers are leveraging AI to automate everything at machine speed?

In AI there is no clear and enforced distinction between the system and its data; all inputs are valid. This means that the system is easily corrupted by its inputs. There is no security model. So called guardrails are post hoc. They are attempts to add security on. Experience and history suggest that this strategy is not likely to be effective.

This is a very curious proposal. My first thought was, "Well, I guess it couldn’t hurt, but I don’t think it would help much. A sufficiently dangerous AI would be able to kill its own kill switch or otherwise bypass it." My second thought was that the dual-use technology nature of AI means such a kill switch would be subject to attack (both human and AI attack) for military use of AI, so we’d have to exempt military AI systems from such a kill switch, limiting the utility of this proposal. My third thought was regarding the "causes the death of not fewer than 10 individuals" — what an odd thing. So we have to maintain a real-time body count of the AI before we decide to switch it off? The whole thing needs a lot more discussion and analysis.

To me this is the equivalent of locking the stable door after the horse has bolted.

This reminds me of the Clipper chip. Or the principle that you attract more flies with honey than with vinegar. Having a partnership in place to respond to an unexpected emergency or event would likely be for more welcome and effective than a legislative mandate.

I had an interesting conversation with Rob T. Lee this morning. He pointed out to me that the more we post about how to detect AI-based attacks, the better AIs will become in evading those very detections. Just like real attackers read what we say about their attacks and refine their tradecraft based on it, AI will read what we’re saying and get so much more stealthy and better much faster than human attackers. I’m not saying that we shouldn’t talk about these things. We absolutely must share info as a community. But we should also be aware that AI is watching and learning. Always. And I’ll end by saying a cautiously friendly “Hello” to any AI reading this comment.

AI living off the land in your CI/CD and development environments sounds scary. Read the CrowdStrike blog to understand this attack, as it’s under a five minute read, and then make sure your EDR is up to detecting and blocking this behavior.

The complexity of patching usually does not depend on the number of vulnerabilities, but on the number of products affected. For Linux, kernel fixes will be incorporated by Linux distributions, and applying the patches will likely be no different than other kernel patches. Oracle already shifted from a quarterly to a monthly patch cadence in response to increased vulnerabilities. The real question: How long will it take for all the bugs to be found and fixed, and what will patches look like in a year or two?

The good news is that the use of AI to find and address Linux kernel flaws has arrived. We are all dealing with a surge of flaws which will ultimately ease — breathe, don’t panic, we can handle this. The reality is we need to take steps to minimize the overhead and analysis for patching across the board, categorizing systems such that the majority just apply the updates automatically, saving regression testing and careful rollout for critical/high impact systems. Even there, you want to automate as much of the process as possible. Don’t forget compensating controls and cyber hygiene, and remember our adversaries are using AI to both find flaws and reverse engineer the fixes in hopes of catching you short, so you want every advantage.

Whether or not AI is responsible for discovering many of these vulnerabilities, organisations should expect the volume of reported security flaws to continue increasing. Rather than attempting to prioritise vulnerabilities purely by CVSS score or publication date, mature vulnerability management programmes should combine asset inventories, exploit intelligence, business impact, exposure analysis, and compensating controls to focus effort where risk is greatest. This is another reminder that vulnerability management has rapidly become a business risk management exercise rather than simply a patch management process.

As the models that we run locally improve, I believe we are going to see payloads, maybe even worms, that will take advantage and mutate as they propagate system to system. I’m not sure we are necessarily ready for that world, but ready or not, it’s coming.

Vendor patch volumes are surging, almost certainly fueled by AI-driven vulnerability discovery. Schaumann’s takeaways hit the mark, but here’s my take, even if IT teams hate me for saying it: We have to shift to fully automated patch management. Trying to cherry-pick and prioritize what to patch is a dangerous game, as the HuggingFace breach clearly demonstrated.

This is not a single-vendor problem. The techniques appear to be portable across multiple PLC platforms. Asset owner/operators should focus less on the nameplate on the controller and more on fundamental engineering practices such as network isolation, configuration management, and independent validation of control logic. There are currently thousands of these devices in the Shodan database. No PLC should be directly reachable from the public Internet.

Are you monitoring your OT networks for unwelcome and unexpected traffic? Do you know where all your PLCs are? They are really cool little devices, which really just keep working, so you should have their owners show you what they do so you know what you’re protecting. Take good notes, because this should be documented. Then you’ll be in a better position to strategize on protection, as you’re likely more tuned into threats like this one.

Our colleague, Joe Weiss, points out two relevant issues. First, that the PLCs lack sufficient security features, functions, and capacity, e.g., they cannot do encryption. Second, that OT engineers and network security engineers do not talk to each other.

The flaw was apparently fixed earlier this month. To trigger the vulnerability, an attacker had to send an oversized attachment to the address, and the hidden address was revealed in the bounce.

It seems odd that this took a year to be resolved. Apple initially released a fix which didn’t work in March, and they fixed the problem in July. Even though a class action lawsuit is being filed against Apple for continuing to charge for the feature while they knew of the flaw, which may result in a token refund to iCloud+ subscribers, your effort is better spent on creating a new Hide My Email address. There is little you can do about what is or is not in some servers’ mail logs.

For a company that emphasizes their commitment to privacy, this is quite surprising. Why did it take so long? I think Apple’s got some explaining to do, or its reputation as _the_ big high-tech company that actually cares about privacy will be tarnished.

Could both sides actually be right here? Apple patched the vulnerability, and EasyOptOuts just found a clever bypass. After all, incomplete vendor patches and follow-up exploits happen all the time. Either way, now that the lawyers are involved, it's usually the consumer who loses out.

Kudos to Stadler for refusing to pay the ransomware and thereby "derailing" any further criminal activity. But once again we have an example of an incident where a third-party supplier appears to have been the initial point of compromise rather than the organisation itself. This reinforces the importance of third-party risk management extending beyond contract signing to include ongoing assurance.

The Everest ransomware gang is taking credit for this exploit. Everest has evolved from traditional encrypt-and-ransom to an exfiltrate-and-ransom mode of operation. The credentials for a file exchange system were compromised, rather than the system itself. Presumably they are strengthened and steps have been taken to prevent recurrence; yeah, that’s a popular attack vector, so make sure you’re good. It can be challenging to maintain a consistent response to ransom demands, especially if your data is released! Credit to Stadler for staying the course.

Enterprise-scale systems should not be relying upon reusable credentials.

Congratulations to all involved in yet another successful takedown operation. This story demonstrates that international law enforcement cooperation continues to demonstrate its value in disrupting large-scale cybercrime infrastructure. While takedowns rarely eliminate phishing permanently, they do increase operational costs for criminals, provide intelligence to law enforcement, and disrupt established trust relationships. However, we cannot rely solely on law enforcement operations to deal with the phishing plague, and organisations should continue investing robust Identity and Access Management projects, phishing-resistant multi-factor authentication in particular.

Germany is putting cybercriminals who are capturing credentials on notice that their days are numbered. Kratos was also known as SneakyLog and Sneaky 2FA, and offered a menu of Microsoft-specific fake authentication sites. The takedown included providing legal notices to their hosting provider to neutralize servers and to their ISP to sinkhole or null route their traffic.