Automated Hunting of Software Update Supply Chain Attacks

Software that automatically updates itself presents an attack surface, which can be leveraged en masse through the compromise of the vendor's infrastructure. This has been seen multiple times during 2017, with high profile examples including NotPetya and CCleaner.

Most large organisations have built robust perimeter defences for incoming and outgoing traffic, but this threat vector is slightly different and far more difficult to detect. Update packages are often deployed in compressed, encrypted or proprietary formats and would not easily be subject to an antivirus scan or sandbox analysis during transit. This leaves us with a large number of trusted processes within our infrastructure that could turn on us at any time and download something evil, which could potentially be undetectable by endpoint antivirus software.

It would be almost impossible to detect all potential malicious code changes, as they could be as simple as changing a single assembly instruction from JNZ to JZ to allow for unauthorised access or privilege escalation to occur. However, this doesn't prevent some additional proportionate due diligence on the update package being pulled down and installed.

Methodology for discovering inbound evil updates

1. Discover all the software across your estate that could be auto updating
Let us consider for a moment software that automatically updates, and how it looks on the endpoint. Firstly, it needs to open an internet connection to a server that will allow it to identify whether the version currently running is up to date, or whether there is a newer version available. This may occur when the process is first executed or at intervals (set dates or times or even randomly). If the version is current then it will try again later, mostly likely connecting to the same server and repeating the process of checking the version numbers. The traffic volume on each occasion is likely to be very small, after all, all that it needs to ask is "what is the current version number?" and the reply need only be "1.0.32" or similar.

If we put these components together we can build a search for a process, other than a browser (to reduce the noise), which makes repetitive connections to the same server on multiple occasions. We can further refine by looking for low volume traffic in the transmissions. This data may contain false positives such as the checking of licence validation, however this doesn't matter as we will refine this further in the next stages.
2. Monitor traffic ratios for evidence of updates being downloaded
Once we have a list of processes, devices and servers that have displayed the behaviour from our discovery phase, we can now monitor the traffic volume ratios for abnormal behaviour. Consider a process that had been polling the same server with 5KB uploads followed by 50KB downloads for the past month when suddenly the volume of the download changes to 300MB. It would be a clear outlier based on volumes and upload/download ratios.
3. Perform automated sandbox analysis on detected binaries post update
Now we're looking for a suspicious update, not just any update, so we can send the installer from the endpoint to an internal sandbox for analysis. Alternatively, we could trigger the same update to occur within a sandbox running our gold image. The sandbox would perform a behavioural analysis, which we would use to highlight any suspicious behaviour during, or after the update process takes place.
4. Alert CERT for further examination should there be evidence of anything suspicious
This can then feed the alerting system used by our SOC/CERT for a manual review and prompt containment activity should the updates contain malware - at this point the number of machines that have pulled the update should be small and we can prevent further infections through proxy blocks etc.

Demonstration of concept

Here we will demonstrate the concept by conducting a deep dive into the behavior of the Notepad++ update process. We begin by identifying the polling between the process and update server. We can see from Process Monitor below that the traffic volumes are low, and the upload / download ratio is relatively similar, approximately 1:11 when there is no update to be downloaded.

Now let's have a look at what happens when there is an update available.

We can see the difference when there is an update to be pulled, in this case the download volume is significantly higher than the upload as we would expect, in fact it's over 673 times higher at 1:7756.

In addition, following the network activity we can see that a process launch has been identified. This is the start of the update installation and provides us details on the location of the update binary. At this point we can take the desired action to review the file, this may include a local scan or better yet, uploading the binary to an internal sandbox for behavioural analysis to take place - this would give us the added benefit of detecting a threat before the antivirus signatures are available.

Practicalities and further ideas

In a large network there is going to be a lot of noise and you may want to select the top 50 auto updating processes that are most widely utilised across the estate and focus on them. This way you can be more selective about the alerts that go to your SOC.

Some processes may update on the fly using hot patching, in which case better results would be obtained from the creation of a custom image for your sandbox with all of these processes installed, so there would be no need to send/pull binaries and you could monitor the entirety of the update process taking place, including all files dropped to disk and executed during the update.

Feedback and ideas are always welcome!

Happy hunting

-Adam (Twitter: @CyberKramer)