Cautionary Tales: Carefully Vet Remote Hires; Ensure Updates are Thoroughly Tested Before Release

This is a good news story: The endpoint protection software detected the malicious activity, and the SOC paid attention and took swift action. Companies may also reconsider remote only hiring. Deep fakes are only getting better and having an in person meeting with a candidate should be required.

Johannes Ullrich

Forward KnowBe4's 'Tips to Prevent This' to your HR Manager, CIO and COO.

John Pescatore

Interestingly they hired this person for an AI position, interviews were virtual and likely using faked imagery. As Paul Asadoorian postulated: "Had they not begun loading malware, I wonder how long they could have worked there and done other things that are not as obvious (like exfiltrate IP)." No data was lost, the attempt to load malware was detected by the laptop EDR, nor is this a breach notification; in this era of hiring workers we may never see in person, this is a learning opportunity. Just how rigorously are you vetting remote hires? Do you challenge remote workers with different work and shipping addresses? Insist on camera on interviews? Require more than just email reference checks? Check resumes for career inconsistencies? Identify conflicting personal information and unexplained unavailability? Your HR folks may be more aware of these risks than you think.

Lee Neely

A potential supply chain attack with an insider twist. With today's largely remote workforce, validating identity is difficult, especially with the use of generative AI. In some organizations a new employee may not visit a corporate office for weeks to months, ample time to create mischief. Kudos to KnowBe4 for disclosing as their tips can be used to guide changes in company hiring processes.

Curtis Dukes

The most important step in IAM is to get the identity right. If one fails in that step, all the authentication down the line will not help. This is true for knowing your customers, employees, partners, vendors, et. al. We tend to focus on fraudulent transactions though fraudulent applications are the greater risk.

William Hugh Murray

This person gets through the process to get hired and, within a week, destroys all their work by trying to subvert their system immediately. Probably not the best operational practice; this could have been much worse.

Moses Frost

Anyone who has written software probably understands this is common. Software bugs happen, edge cases happen, and the question now becomes, why did we have a single point of failure in so many systems? Maybe that should be our question, not how a software bug happens. This could have easily been Microsoft Defender ATP.

Moses Frost

Keep in mind that this was Rapid Response Content, the code we want quickly to thwart active exploit techniques. This isn't content you can stagger and delay, as you can sensor updates. Also, CrowdStrike had an existing suite of automated regression and stress testing processes. Unfortunately, there was a bug in one of the validators allowing the flawed code to be released. To address these issues CrowdStrike is both improving their testing/QA processes and creating greater control for the deployment of Rapid Response Content updates as well as content update details in release notes for customer review. Consider these changes when the topic of future plans for CrowdStrike are discussed, ripping and replacing would be a bad idea.

Lee Neely

The unfortunate consequence of this outage is that organizations will look to delay the patching/updating of their systems. Patching is already difficult enough without the added pressure that it could take a network down. Any delay in updating systems only gives the adversary extra time to develop exploits and prosecute the attack.

Curtis Dukes

We knew within hours that there was a quality assurance failure that contributed to this outrage. Experienced people also knew that there would be plenty of blame to go around. Included among the many contributing decisions was Microsoft's strategy of not breaking legacy systems, and its expedient decision a decade ago to grant CrowdStrike and its competitors access to ring zero rather than providing an API, and then never revisiting that decision. While there is no single cause or remedy, there is an observation: Microsoft, CrowdStrike, and our IT management culture have outgrown the reasons for their success. This outage should be a wakeup call but there is no easy fix.

William Hugh Murray

Modern 911 systems are a good example how complexity affects resilience. The integration of VoIP and other messaging modes, as well as more complex routing due to mobile phone and VoIP customers have made 911 outages a somewhat regular occurrence.

Johannes Ullrich

The outage, which resulted from improperly testing an update, affected 125 million devices. Beyond the root cause of not following AT&T procedures to test the change prior to deployment, the outage, which caused the network to enter "protect mode" to protect other services, was protracted (the change was rolled out within 2 hours) by an overwhelming volume from devices attempting to re-register themselves on the network. The problem is that our modern IT, whether in a large shop like AT&T or your own, has an incredible number of interdependencies, internal and external, necessitating increased rigor of change management and regression testing, which is really hard for someone like me who wants to move rapidly. AT&T faces regulators and possible fines; Verizon is paying $1M for a December 2022 outage in six states of 104 minutes. While your next outage may not have regulatory consequences, your users will be just as adamant in expressing their appreciation of the inconvenience, a situation we all want to avoid.

Lee Neely

The conclusion by the FCC that AT&T did not adhere to industry-developed best practices is probably the most damning. In other words, AT&T didn't exhibit a Ôstandard of reasonablenessÕ in managing their network. Plaintiff attorneys are starting to use terms like 'failure to implement adequate and reasonable cybersecurity procedures' in court filings. One can expect this outage also to be litigated.

Curtis Dukes

These policies put Crowdstrike in line with other similar products, who learned the lesson during similar incidents.

Johannes Ullrich

CrowdStrike has acknowledged the need to add more controls on content updates, and is rolling out settings you can adjust, as well as adding content update details to release notes you can subscribe to.

Lee Neely

For most of us, the safe default is to enable automatic updates. For us, the risk of not being current is greater than that an update will damage us. For large dependent enterprises, like banks and airlines, not so much. The more devices and mission critical applications involved and the closer to the hardware, the more caution must be taken.

William Hugh Murray

There don't appear to be effective workarounds for these flaws other than updating to the fixed version of BIND. While you're out checking and updating your BIND installations, it'd be a good time to ask your DNS team for their plans on DoH/DoT as you really want an enterprise approach here, rather than mixed results-based on endpoint product implementations of these protocols.

Lee Neely

I'm going to skew almost 50 years old-time here, feel free to skip: back in 1975, Saturday Night Live Weekend Update had a recurring bit where Chevy Chase would say: 'In breaking news, Generalissimo Francisco Franco is still dead!' Five years later, in 1980 or so, the first version of the BIND DNS software came out. Now 44 years after that release high severity vulnerabilities are still being found in BIND. Breaking news: Software 'engineering' is largely an oxymoron; each new release is still an adventure - as CrowdStrike's bad update (and even worse testing software) certainly reinforced.

John Pescatore

No one wants to touch DNS.

Moses Frost

CVE-2024-41110, Docker authentication bypass, CVSS score of 9.9, warrants attention. The complexity of this attack is low. The patches have been merged into the master, 19.0, 20.0, 23.0, 24.0, 25.0, 26.0 and 26.1 release branches; Docker recommends versions greater than v23.0.14 or v27.1.0. The primary fix is to update your containers, a workaround is to avoid using AuthZ plugins and/or restrict access to the Docker API to trusted parties only. Given the ease of quiescing a container and deploying a new one, skip the workaround and update your containers.

Lee Neely

CVE-2024-37998 allows an attacker to reset the password of admin accounts without knowing the current password when auto-login is enabled. Siemens identified disabling auto-login as a workaround. Check for operational impact before toggling that option; applying the update may be the lower risk option. CVE-2024-39601 allows an attacker to downgrade the device firmware, or execute arbitrary commands. The fix here is to apply the update. Even with proper segmentation and access controls, you want to get the flawed software out of the system in case missed a potential exploit path.

Lee Neely

The tech industry is attractive for several reasons including the pressure to deliver, which increases the likelihood of a ransom payout, other sectors dependencies on their products (think supply chain and third-party), and rapid adoption of technology which may have flaws (think Snowflake). There is no silver bullet here, except to remember the basics: keep services patched, validate security measures, particularly on new services, use MFA, and monitoring.

Lee Neely

Sounds like an astronomy related social network, doesn't it? Sadly, this is far less lofty and more malicious. This gang has two novel tactics. First is phishing without email - posting links instead on Discord, Twitch, Instagram, YouTube, X (Twitter), Trovo, TikTok. Clicking that links takes users to the second tactic: benign looking GitHub accounts that trigger a three-stage attack, tricking the victim into accessing/installing the loaded archive. While GitHub is deactivating any accounts identified with this sort of effort, these same techniques will work on other delivery platforms. Beware of links in online discussions as well as look twice before accessing links in a README.md, particularly those which lead outside the repository.

Lee Neely

The ransom was paid into a bitcoin account, which was then transferred into addresses belonging two tow Hong Kong residents, where it was converted into Chinese currency, transferred to a Chinese bank, then accessed from an ATM in China next to the Sino-Korea friendship bridge. While the indictment is unlikely to result in an arrest, it could result in sanctions which make it harder for ransomware payments to be collected/laundered in this fashion in the future.

Lee Neely

It's highly unlikely the individual will ever be arrested. It's also unlikely that this indictment will change North Korea's unstated policy of using cyber-attacks to fund the regime.

Curtis Dukes