Rackspace Outage Emphasizes Need for Cloud Outage Workarounds; Check for Falsely Signed Malicious Android Apps; Isolate Baseboard Management Controllers from External Connectivity

Last week I gave a lunchtime talk on Capitol Hill to Congressional aides who were involved in writing cybersecurity policy and one of the questions was “Wouldn’t all of this be solved if everything were run in the cloud, like Netflix and Amazon?” After all, a short “spinny circle of death” delay in video seems much better than what happened at the Colonial Gas Pipeline… Good to use example like Rackspace’s woes to make sure backup plans are in place and tested and that management understands that security issues really don’t change that much whether the computers are in our buildings or in the cloud’s buildings.

John Pescatore

Rackspace is providing support to either migrate to MS 365 or forward your email to another domain. Migrating to MS 365 is going to be the most familiar option, and Rackspace is providing archive copies of inboxes to customers for import into MS 365. Note that with either option there may be email "in flight" which may need to be resent as it is queued and waiting to be delivered. When migrating make sure you implement needed security settings such as MFA, ATP, leverage the Microsoft 365 Defender and Microsoft Purview compliance portals to make sure you aren't missing anything.

Lee Neely

This may be one of the most interesting security incidents in a while. Rackspace’s business model is in reselling its hosted solutions. In this case, they have done what, in my opinion, is the right thing. They have started to request customers move over to the Microsoft 365 service. Rackspace has possibly a better change of rolling out patches quickly in their environments, but let’s face it, Microsoft is more in control of the source code of exchange than we are, and they may even start rolling out patches before anyone else.

Moses Frost

Not a good day, week, or month to come for Rackspace. Hopefully, once systems have been restored and user operation back to normal, Rackspace will fully share details of the event – to include what security applications were in place and operating. We all can learn from this unfortunate cyber incident.

Curtis Dukes

This story is sad for a number of reasons: First of all, there appears to be no good way to replace the certificates or revoke them. The only thing that apparently can be done is to not allow applications signed with these certificates in the Google play store. Secondly, the malware signed with these certificates has been circulating for years. It was just recently discovered that the certificates used are Android platform certificates. Third: There is no word from the affected companies about how the certificates ended up being used to sign malware. And... the certificates will not expire until well into the 2030s.

Johannes Ullrich

Platform certificates are kinda like skeleton keys that can open every lock (OK, even I’m not old enough to ever have actually used a skeleton key…) Their use should obviously be minimized, and they need to be really, really protected. Looks like neither of these principles were followed by the Android OEMs. The fact that multiple OEMs offer Android phones and tablets has helped Google gain market share over Apple but this (and the “patch gap” issues) point out the risks and the fact that Google needs to invest more in making sure the Android ecosystem is trustable.

John Pescatore

The platform certificate allows the OS to run with high levels of privilege, so any applications signed with those stolen credentials can also run at those increased privilege levels without user consent. OEM's have been pushing out updated certificates over the air; make sure that you keep your devices updated to capture these new certificates. Train users to review installed applications, removing those they don't use, and revisiting/reducing the permissions on those they do to.

Lee Neely

Without going into much detail, we have seen code-signing certificates in the wild left in an unsecured state. This is the worst-case scenario for many platforms. The operating systems use certificates as trust anchors, so make sure you properly secure them.

Moses Frost

Certificates are public meta data about public keys; copying them is not "stealing." They cannot be used to sign. It is the private half of the key-pair that is used to sign and that has been stolen. Signing keys should not be kept online when not in use.

William Hugh Murray

Part of the issue is "redfish", the new API meant to replace IPMI. Redfish is based around "web standards" which apparently means that we now include standard web application vulnerabilities like OS command injection in BMC software. These days, applications are web applications. If it is a BMC, a mobile app or a word processor. You will only be able to defend your organization if you understand web applications.

Johannes Ullrich

If you're not already doing so, consider the BMC as equivalent to standing at the physical console of the system. The services enable your system administrators to do almost anything from wherever they are located. As such, you really need to restrict the access to only users and devices that need to access them. Never expose these directly to the Internet. Now it gets harder - you need to keep them updated, make sure that you're only running the genuine/vetted versions, and monitor for anomalous behavior. Make sure that you have a non-production system to test updates, as you can effectively kneecap a system getting this wrong.

Lee Neely

The hospitals are moving patients, in part, because the automated/connected monitoring systems are inoperable, and it takes a substantial increase in resources for manual monitoring. They are also wisely choosing to not initiate services they cannot fully support. When thinking about an attack which takes your IT systems offline, don't casually plan to revert to manual methods: make sure you've done a deep dive on not only what manual means, but also the increased staff and lowered throughput in that scenario. Factor in what can be delayed or redirected. In the early 1980s, I was working my way through college in retail. With turnover, I became the only one in my district who knew the manual methods when the computerized system failed, including having a supply of the forms for manual reporting. Make sure that you have training and references so staff can successfully adapt, avoid having a single point of expertise.

Lee Neely

Three points to be made here: 1) the healthcare sector continues to be a primary target of cyber criminals looking for a quick payout; 2) connectivity of operational technology, in this case patient monitors, with IT systems can disrupt business operations; and, 3) each cyber breach that is reported serves as a warning to the executive team to revisit cyber defense plans that include knowing their environment [HW, SW, Data], configuration management, vulnerability management, account management, and network monitoring of their enterprise.

Curtis Dukes

The ransomware epidemic will probably not be over anytime soon. The culture in many healthcare organizations prioritizes patient safety over other initiatives such as “secure computing.” I know that this is probably concerning to many folks reading this editorial. However, the fact is that patient safety and computer safety have not historically been tied together in a clinical setting. The last half decade of these attacks may start shifting these attitudes. The more clinicians rely on these systems for patient safety, the more healthcare organizations will need to take a different approach to their internal systems. Unfortunately, if the HealthCare IT community doesn’t resolve this, it will be resolved by regulation.

Moses Frost

Hospitals continue to be favorite targets of ransomware attacks, in part because clinical applications are so sensitive. These applications should be isolated from those, like browsing and e-mail, that use public networks.

William Hugh Murray

GAO almost lost me in the Executive Summary when they used one of the dreaded null value words, “holistic” -“heuristic” is the other one. But they did point out that as of 4 December 2022, Government agencies can’t buy IoT devices that don’t meet NIST standards that most don’t meet, and that OMB had failed to define a waiver process. Even some of the most impactful government procurement restrictions (like requiring FIPS 140-1 compliance in all procurement of cryptography) require at least a temporary waiver in order to get going and have actual effect on markets. On 2 December OMB issued waiver guidance (https://www.whitehouse.gov/wp-content/uploads/2022/12/M-23-03-FY23-FISMA-Guidance-2.pdf) that requires CIOs to justify not meeting the new regulations. OMB did not require waivers be reviewed or contain any “sunset” provision, so the long-term forecast is for blizzards of waivers and little increase in IoT device security.

John Pescatore

Developing metrics to measure the effectiveness of OT/IoT cyber security can be complicated by their isolation. Currently, federal agencies secure systems using a risk-based approach, driven from the NIST Risk Management Framework, which includes OT and IT systems. Odds are your OT administrators are not familiar with your security framework but are actually aware of an OT framework such as the Purdue model. Take the time to work with them to crosswalk the two before proposing changes so you can come from a common understanding. You are likely already on the same page about the end goals, you just need to align the details and document how you're getting there, which could help you develop metrics and measurements of your success.

Lee Neely

Worthy but difficult goals. With much of IT, we can outsource to large providers. No one needs to secure their own mail server anymore, for example. With all the OT that makes critical infrastructure so critical, there's no easy way to consolidate defensive efforts.

Christopher Elgee

Creation of a common set of metrics, across all industry sectors, is a good thing. When it comes to measuring cybersecurity best practices, each sector has more in common than not.

Curtis Dukes

The Cyber Security Review Board was supposed to follow the model of the National Transportation Safety Board but in its first two efforts the CSRB has diverged in a big way. The NTSB investigates incidents, not vulnerabilities, and eventually got the power to enforce changes. The first CSRB effort produced a great report on Log4j vulnerabilities and risks and had great recommendations for change – but there were already plenty of those out there. The CSRB could never investigate every incident, but it wouldn’t have to. The focus on what went wrong that that enabled something like the Colonial Gas Pipeline gasoline supply chain disruption and driving legislation to prevent it from happening again is what is needed. I hope this one focuses on a particular attack by Lapsus$ vs. a report on the group’s tactics overall.

John Pescatore

The CSRB topic is moving from analysis of a vulnerability that affected millions of organizations to that of a highly skilled threat actor that targets specific organizations. The cybersecurity community looks forward to better understanding what cyber defenses were in-place, what security controls failed, and incident response techniques employed.

Curtis Dukes

As the CSRB finds its voice and process, documents like this upcoming report will be a good reference to both understand and defend against these types of attacks. Timing will be the trick: having these reports while the threat is imminent will dramatically increase their usefulness.

Lee Neely

I can’t wait to read this one. AKA, I’m very surprised that 16-year-olds using relatively unsophisticated techniques have gotten as far as state actors. There was nothing “novel” about what they did, yet they did it anyway.

Moses Frost

Note there is no indication that the Cuba Ransomware actors have any affiliation or connection with the Republic of Cuba. The main targets are financial services, government, healthcare and public health, critical manufacturing, and IT. The attackers use known bugs (CVE-2022-24521, CVE-2020-1472), phishing, compromised credentials and RDP to gain access, then use the Hancitor loader to drop their ransomware. Count this gang in the double ransom category - one payment to decrypt, another to not post your data. Grab the IOCs of the CISA alert, make sure that you're really doing comprehensive MFA on anything Internet accessible, track patching/updating, prioritizing boundary protection, remote access and Internet facing services. Take a breath, now go back and make sure that your SOC is actively monitoring and responding to abnormal activity, make sure that you're enabling available security alerts such as impossible logins, refresh your coffee and review the alert for anything else you can do. Hopefully this is all already in place.

Lee Neely

Fast forward a year from initial warning of this ransomware gang: technical details updated; indicators of compromise updated; TTPs mapped to Mitre ATT&CK; and mitigations updated. Meanwhile the gang continues to exceed its yearly business objectives. The only way to ‘stop ransomware effort’ is to automate the processes around configuration and vulnerability management.

Curtis Dukes

We should be resisting ransomware, not looking for it. The window between the initial breach and the success of ransomware is short and shrinking. One is not likely to detect it in this window. At the end of the window, it will announce itself.

William Hugh Murray

Two things to think about when reading this. #1: Adobe Reader (that one probably stings). #2: Our web browsers are designed to download and compile/run code from 3rd parties. These are very complicated systems; add to that, we are now seeing more and more bugs due to how much research is being built up; expect more. This is one of the reasons Mozilla started to invest heavily in developing rust, as they also faced the same issues.

Moses Frost

Again we're dealing with a weakness in the Chrome V8 JavaScript engine, which means you're going to be updating Chrome and Chromium based browsers. I know you're thinking about disabling JavaScript, which, while noble, is not truly viable with the plethora of web based applications in use today. Rely on EDR and boundary protections to reinforce browser security for that defense-in-depth approach.

Lee Neely

While nine zero-day vulnerabilities in a year is a lot, I suspect most can be attributed to Google’s internal threat analysis group. The really good news is that Google greatly simplified the patch management process for Chrome, now mirrored by all major browser vendors. It’s as simple as closing and reopening the browser.

Curtis Dukes

This is not a ransomware strain you can decrypt to recover from: you're going to need those differential backups we've been discussing. Today, this malware is highly targeted, focusing on Russian mayor's offices and courts, and we know that can change, so incorporate known IOCs into your threat hunting activities. This, like others, spreads through network weaknesses as well as email attachments, so make sure you're monitoring your network, filtering URLs and attachments in email to the extent possible, as well as providing guidance to users on link and attachment handling.

Lee Neely

Consider "read only" and "execute only" access control rules to reduce the potential risk of both ransomware and wipers.

William Hugh Murray

With the rate of data flowing into centralized logging, you need every trick to identify anomalous behavior, while eliminating both false positives and negatives. The point is that thresholds, our old standbys, aren't sufficient in today's environment. You need mechanisms that adapt and learn based on events seen. If you already have tools intended to do this, make sure they are enabled and configured; it's going to take time to get them tuned/trained, but you may find you discover anomalous behavior previously overlooked.

Lee Neely