Cautionary Tales: Carefully Vet Remote Hires; Ensure Updates are Thoroughly Tested Before Release

This is a good news story: The endpoint protection software detected the malicious activity, and the SOC paid attention and took swift action. Companies may also reconsider remote only hiring. Deep fakes are only getting better and having an in person meeting with a candidate should be required.

Forward KnowBe4's 'Tips to Prevent This' to your HR Manager, CIO and COO.

Interestingly they hired this person for an AI position, interviews were virtual and likely using faked imagery. As Paul Asadoorian postulated: "Had they not begun loading malware, I wonder how long they could have worked there and done other things that are not as obvious (like exfiltrate IP)." No data was lost, the attempt to load malware was detected by the laptop EDR, nor is this a breach notification; in this era of hiring workers we may never see in person, this is a learning opportunity. Just how rigorously are you vetting remote hires? Do you challenge remote workers with different work and shipping addresses? Insist on camera on interviews? Require more than just email reference checks? Check resumes for career inconsistencies? Identify conflicting personal information and unexplained unavailability? Your HR folks may be more aware of these risks than you think.

A potential supply chain attack with an insider twist. With today's largely remote workforce, validating identity is difficult, especially with the use of generative AI. In some organizations a new employee may not visit a corporate office for weeks to months, ample time to create mischief. Kudos to KnowBe4 for disclosing as their tips can be used to guide changes in company hiring processes.

The most important step in IAM is to get the identity right. If one fails in that step, all the authentication down the line will not help. This is true for knowing your customers, employees, partners, vendors, et. al. We tend to focus on fraudulent transactions though fraudulent applications are the greater risk.

This person gets through the process to get hired and, within a week, destroys all their work by trying to subvert their system immediately. Probably not the best operational practice; this could have been much worse.

Anyone who has written software probably understands this is common. Software bugs happen, edge cases happen, and the question now becomes, why did we have a single point of failure in so many systems? Maybe that should be our question, not how a software bug happens. This could have easily been Microsoft Defender ATP.

Keep in mind that this was Rapid Response Content, the code we want quickly to thwart active exploit techniques. This isn't content you can stagger and delay, as you can sensor updates. Also, CrowdStrike had an existing suite of automated regression and stress testing processes. Unfortunately, there was a bug in one of the validators allowing the flawed code to be released. To address these issues CrowdStrike is both improving their testing/QA processes and creating greater control for the deployment of Rapid Response Content updates as well as content update details in release notes for customer review. Consider these changes when the topic of future plans for CrowdStrike are discussed, ripping and replacing would be a bad idea.

The unfortunate consequence of this outage is that organizations will look to delay the patching/updating of their systems. Patching is already difficult enough without the added pressure that it could take a network down. Any delay in updating systems only gives the adversary extra time to develop exploits and prosecute the attack.

We knew within hours that there was a quality assurance failure that contributed to this outrage. Experienced people also knew that there would be plenty of blame to go around. Included among the many contributing decisions was Microsoft's strategy of not breaking legacy systems, and its expedient decision a decade ago to grant CrowdStrike and its competitors access to ring zero rather than providing an API, and then never revisiting that decision. While there is no single cause or remedy, there is an observation: Microsoft, CrowdStrike, and our IT management culture have outgrown the reasons for their success. This outage should be a wakeup call but there is no easy fix.

Modern 911 systems are a good example how complexity affects resilience. The integration of VoIP and other messaging modes, as well as more complex routing due to mobile phone and VoIP customers have made 911 outages a somewhat regular occurrence.

The outage, which resulted from improperly testing an update, affected 125 million devices. Beyond the root cause of not following AT&T procedures to test the change prior to deployment, the outage, which caused the network to enter "protect mode" to protect other services, was protracted (the change was rolled out within 2 hours) by an overwhelming volume from devices attempting to re-register themselves on the network. The problem is that our modern IT, whether in a large shop like AT&T or your own, has an incredible number of interdependencies, internal and external, necessitating increased rigor of change management and regression testing, which is really hard for someone like me who wants to move rapidly. AT&T faces regulators and possible fines; Verizon is paying $1M for a December 2022 outage in six states of 104 minutes. While your next outage may not have regulatory consequences, your users will be just as adamant in expressing their appreciation of the inconvenience, a situation we all want to avoid.

The conclusion by the FCC that AT&T did not adhere to industry-developed best practices is probably the most damning. In other words, AT&T didn't exhibit a Ôstandard of reasonablenessÕ in managing their network. Plaintiff attorneys are starting to use terms like 'failure to implement adequate and reasonable cybersecurity procedures' in court filings. One can expect this outage also to be litigated.

These policies put Crowdstrike in line with other similar products, who learned the lesson during similar incidents.

CrowdStrike has acknowledged the need to add more controls on content updates, and is rolling out settings you can adjust, as well as adding content update details to release notes you can subscribe to.

For most of us, the safe default is to enable automatic updates. For us, the risk of not being current is greater than that an update will damage us. For large dependent enterprises, like banks and airlines, not so much. The more devices and mission critical applications involved and the closer to the hardware, the more caution must be taken.

There don't appear to be effective workarounds for these flaws other than updating to the fixed version of BIND. While you're out checking and updating your BIND installations, it'd be a good time to ask your DNS team for their plans on DoH/DoT as you really want an enterprise approach here, rather than mixed results-based on endpoint product implementations of these protocols.

I'm going to skew almost 50 years old-time here, feel free to skip: back in 1975, Saturday Night Live Weekend Update had a recurring bit where Chevy Chase would say: 'In breaking news, Generalissimo Francisco Franco is still dead!' Five years later, in 1980 or so, the first version of the BIND DNS software came out. Now 44 years after that release high severity vulnerabilities are still being found in BIND. Breaking news: Software 'engineering' is largely an oxymoron; each new release is still an adventure - as CrowdStrike's bad update (and even worse testing software) certainly reinforced.

No one wants to touch DNS.

The ransom was paid into a bitcoin account, which was then transferred into addresses belonging two tow Hong Kong residents, where it was converted into Chinese currency, transferred to a Chinese bank, then accessed from an ATM in China next to the Sino-Korea friendship bridge. While the indictment is unlikely to result in an arrest, it could result in sanctions which make it harder for ransomware payments to be collected/laundered in this fashion in the future.

It's highly unlikely the individual will ever be arrested. It's also unlikely that this indictment will change North Korea's unstated policy of using cyber-attacks to fund the regime.