CrowdStrike Outages: What Happened?; Remediation Guidance; Microsoft's Recovery Tool; Consolidation Risks

Currently, some organizations are still struggling to recover. But over the next couple weeks, I expect CrowdStrike to openly discuss what process failed and how they are going to address this in the future. In June, similar issues appeared to have happened with select Linux systems. The issue was not as wide spread, and did not have the same impact as a result. But it appears that CrowdStrike must look at their configuration file build process to avoid future issues like this.

Johannes Ullrich

CrowdStrike founder and CEO has committed to full transparency on how this occurred and steps we're taking to prevent anything like this from happening again. That is important to evaluate and track if you are considering renewing or selecting CrowdStrike use.

John Pescatore

CrowdStrike had an issue, but will companies lift and shift their ops over this event? I am not so sure. What would be more pragmatic is for these machines that caused drastic issues to be evaluated. Can there be a redundant system? Could that system have a different EDR? That type of diversity is better than just a single system point of failure. There are pros and cons to all of this.

Moses Frost

Consider this a bad malware definition update; remember those are pushed as rapidly as possible to protect us from harm. In this case, a newly observed C2 frameworks which utilized named pipes. At this point, I'm predicting CrowdStrike will have an industry leading QA process to insure this never happens again.

Lee Neely

I sincerely hope that CrowdStrike publishes a full and transparent report on what happened and what measures they are putting in place to prevent a re-occurrence so that other vendors in the industry can improve their own processes and customers may be able to implement mitigating controls.

Brian Honan

The part that CrowdStrike did well was to offer detailed and continuously updated recovery instructions. Initially, these instructions required a valid login, but CrowdStrike very quickly made them public which helped as many IT workers involved in recovery were not CrowdStrike admins and did not have access to the customer portal.

Johannes Ullrich

Only use remediation guidance from CrowdStrike to remediate items. I know it's tempting to use a third party, but it's best to use the right sources.

Moses Frost

If you're a CrowdStrike shop, odds are your helpdesk had a line with laptop recoveries while other parts of your IT staff likely spent the weekend returning servers to service. Be careful to check any "quick fixes" as adversaries are pushing out alternative solutions which are not what they appear. CrowdStrike also offered an option, if requested, to mark the bad update for your enterprise which would allow systems which were online briefly to possibly self-remove the offending file. You're going to want to leverage all the legitimate options.

Lee Neely

Even if you are not a CrowdStrike customer and were not directly impacted by this outage, I recommend reviewing this guide, together with the USB recovery tool from Microsoft, and adopting it to your own business continuity plans as to how you would recover end point devices in the event of a major issue impacting them.

Brian Honan

The tool provides two recovery mechanisms, either WinPE or safe mode. WinPE doesn't require an admin account but may require manual entry of BitLocker recovery key. Safe mode works if you're using TPM only BitLocker, but if you're using TPM+PIN you need the PIN or Bitlocker recovery key. You'll also need an account with administrator rights to access the system. If you're using FDE, other than BitLocker, you're going to need to use that providers recovery mechanism.

Lee Neely

Although the fault for this global outage lies squarely at CrowdStrikeÕs feet, Microsoft also realizes that its large market share at the endpoint exposes it to risk. In fact, several news stories had Microsoft as the lead caption. Fielding a recovery tool is one way of limiting that risk, while reducing the negative press and calls for technology diversity.

Curtis Dukes

Consolidation and "mono culture" is not a new concern. Dan Geer warned of it back in 2003, but we still prefer the convenience of mono cultures to the resilience of diversity. https://ccianet.org/wp-content/uploads/2003/09/cyberinsecurity%20the%20cost%20of%20monopoly.pdf: Cyber Insecurity: The Cost of Monopoly

Johannes Ullrich

We know that at the extremes, a monoculture and unmanageable chaos are high risk. But, the safest middle ground is really driven by your IT organization and your security organization's ability to manage software/services. For many (really most), 'put most of your eggs in one basket and really, really watch that basket' is an inescapable reality - as Windows market share of 73% proves! Supply chain security does need concentration analysis, as having redundant providers who both use AWS cloud services or in this case were both running CrowdStrike means not so much added value.

John Pescatore

While all security practitioners, me included, preach the need for technology diversity, the reality is that it comes at a cost in system management, monitoring, and skills. Organizations must weigh those costs when selecting the IT and security ecosystem. Oh, and don't forget, resilience and redundancy while lofty words for everyone to strive for in the security business, also add to that cost. For the foreseeable future, this sort of risk will remain, notwithstanding platitudes from politicians and government officials.

Curtis Dukes

Concentration risk is a concern. The question is which is more concentrated, the use of a common EDR (CrowdStrike), or the use of a single OS (Windows?) It may not be practical to change your stance on OS use; you should evaluate the tradeoffs and understand the risks. While CrowdStrike impacted about 8.5 million Windows endpoints, alternate EDR such as Carbon Black, Trend Micro, or even AV solutions have similar numbers of endpoints. Unlike the old model of deploying more than one protection on the endpoint to improve coverage, the risk here is that your protection takes out your system. Many have forgotten that about 25 years ago an AV product marked Microsoft Office as malware and quarantined it. While a smaller scale, the net is the same. Consider that a provider who has been through a scenario like this may be better prepared to both prevent and respond to a future event.

Lee Neely

We have understood the risk of such heavy reliance on one operating system, monoculture, for more than a generation. Collectively we have made the decision to accept the risk. The resistance to revisiting this decision, simple friction, suggests that we will continue to accept the risk. We must reap what we sow.

William Hugh Murray

Fortunately the court was able to detect and respond to contain and minimize this incident. Consider a tabletop where you're not only seeing a ransomware attack but also a secondary incident which also is disrupting services. The goal is to not be distracted by one big event thereby missing another.

Lee Neely

This case has two arguments: 1) authority of the DHS CIO; and 2) mission criticality of software. For the first, it seems logical that a Department CIO would have authority over all department components; else, why even have a department level CIO? For the second, while a component may accept a risk, mitigations should be in place to reduce risk; that didn't happen in the year following the order. It's now on the department leadership to act.

Curtis Dukes

The DHS CISO revoked the authorization to operate (ATO) for the training system, which was deemed unable to protect PII, immediately followed by an approved exception to continue operations, without addressing the identified risks. Revoking the ATO, effectively turning off, a business system is very impactful, and difficult to enforce (ask me how I know this); even so, that process must include a statement about what must be done to resume operations as pressure to resume operations is intense, and you're going to need support from high levels no matter how it plays out.

Lee Neely

At least two other members of the Scattered Spider ransomware gang, including the suspected leader, were arrested a month ago in Spain. In addition to the MGM attack a year ago, Scattered Spider also hit about 100 other organizations, many of which paid the ransom. Here is where involving the FBI in any ransomware attack you face can help in the long run. Make sure that you not only have their local office number, but that you've met their agents and that they know who you are.

Lee Neely

These two are the Russian hacktivist group Cyber Army of Russia Reborn (CARR) leader and primary hacker. CARR initially focused on DDoS attacks directed at Ukraine and has more recently started focusing on US and European critical infrastructure. While the attacks are categorized as unsophisticated, they are still successful. This means you need to check the basics on your ICS/SCADA systems. Don't expose them directly to the Internet, employ segmentation, monitor for unsolicited activity, make sure you're on current best practices to protect these systems.

Lee Neely

While identifying the culprits of attacks on critical infrastructure is important, the practical effect in this case is minimal. It's unlikely they will leave or be handed over by Russian authorities, but they are at least now on a global watch list.

Curtis Dukes

While the attack was discovered July 15th, Cadre Holdings is still in the midst of the investigation and isn't disclosing the extent or impact of the event. In fact, the only reference, from Cadre Holdings to the event is the Form 8-K SEC filing. At the point you're filing a notice with your regulators, you should also be preparing information for your staff, members, etc.

Lee Neely

Well, the rule is not working as intended, i.e., requiring management to inform investors of materiality, but it is encouraging reporting.

William Hugh Murray