If you don't know Hal Pomeranz through his teaching at SANS Institute, contributions to the Command Line Kung Fu blog or postings to this Computer Forensics blog, you've been missing out. Hal's expertise spans several areas of information security, and most recently has focused on forensics. I have the pleasure of speaking with Hal periodically, in part because we both teach SANS' Reverse-Engineering Malware course. We recently talked about malicious software in the context of data breaches and incident response. Hal allowed me to publish his thoughts on these topics in the interview format below.
Lenny: People talk about malware nowadays much more than they did, say, 2 years ago. This seems to be the case not only for infosec pros, but also for the general public. Why do you think that is the case?
Hal: The glib answer is that malware-facilitated attacks have been "in the news" so much that malware has become one of the threat-vectors on everybody's mind. It was probably Google's public acknowledgement of the Operation Aurora breaches that really pushed the matter into the public consciousness. But you could also point to any of the large payment card breaches in the last few years, or the ongoing ACH/banking fraud via Zeus and other malware. All of these directly impacted lots of people outside the InfoSec community.
The historical trend here is that cyber crime has become a business. The attakers want to maintain access on the computers they penetrate in order to use computing resources for tasks like spamming, stealing credentials, moving laterally around the network, and exfiltrating data they acquire. In order to do this, they need malware that
provides remote access, stealth capabilities, and resilience against administrator intervention. It's hard to imagine an attack scenario these days that doesn't have some malware component.
Lenny: How are infections of Unix systems different from those of Windows ones, perhaps in terms of the artifacts left behind, infection vectors or attacker objectives?
Hal: Putting Android aside, Unix and Linux are still predominantly server operating systems. We're not seeing a lot of Linux on the desktop, so you don't often get the "click this link" or drive by web malware that you do in the Windows world.
So the infection vector on the Unix/Linux side is typically still going to be direct compromise. Brute force attacks against services like SSH are still depressingly effective. Or memory corruption attacks (buffer and heap overflows, format string attacks, etc) against standard services, like the recent FreeBSD encrypted telnet exploit. And since there are a lot of web servers running on Unix, you get attackers exploiting poorly written web apps: WordPress, Joomla, PHPMyAdmin, PHPMyBB, et al.
In the case of a web app compromise, you may simply have attackers installing code to advertise their online pharmacy sites. This happened to the Joomla-based site of an organization my wife is affiliated with. Tom Liston over at the Internet Storm Center has written about these kind of attacks in great detail.
Compromised web sites are also used to serve malware to infect unsuspecting Windows users as well. So once again we see that Unix machines make the best servers for Windows, even if what they're serving is malware.
In the case of a full-on root-level operating system compromise on Unix, things really aren't so different from Windows:
- Attackers are going to need some sort of persistence mechanism. On Linux this typically looks like modifying the code that gets loaded when the system boots, whether it's through modification of the scripts that start various system services or modifications to the lower-level boot configuration files.
- Some combination of remote control, spamming, sniffing, and relaying tools will be installed.
- Rootkit tools will be installed to hide the attackers' presence from administrators and resist removal of the malicious software. The relatively low "market share" of Linux/Unix has meant that attackers aren't spending a lot of time on this right now, so their tools are still fairly primitive. But a really well done kernel-level rootkit on Linux has a lot more control over what goes on in the OS-filesystem, processes, and network- than a Windows LKM would in general.
There was recent speculation, based on photos in a unclassified briefing, that the analysis stations for the US Predator drone fleet are moving to Linux. I quipped on Twitter, "Impending Linux malware onslaught in 3... 2... 1..." But it's true that the more critical and sensitive platforms move to Linux and Unix, the more time attackers will spend improving their attacks in this area.
Lenny: A surprisingly high number of strong PDF analysis tools have appeared in the recent years. Why do you think that is the case? Why are there so few tools for examining Microsoft Office documents?
I'm a firm believer that tools get created out of real needs. So we have to ask ourselves why would we be needing better PDF tools? The obvious answer is that we're seeing more PDF-based attacks. Why would attackers prefer PDF over MS Office documents? I would speculate that if I'm an attacker looking for the largest pool of vulnerable machines, PDF might be preferrable to MS Office. Typically, every business PC is going to have MS Office + Adobe Reader. But home machines might not have Office installed. And home machines might be a more tempting target for attackers since they're typically less well monitored and maintained. Once inside, attackers are probably going to be able to hang around for longer on the standard home PC. With home broadband rates commonly in the 12-50Mbps range in major US markets, that's a useful system for an attacker to own.
We could also be optomistic and suggest that Microsoft's internal secure coding initiatives are bearing fruit, and they're general producing better code with fewer defects in each revision of Office. Certainly the regular "Patch Tuesday" update cycle makes it more likely for users to get fixes for bugs in Windows and MS Office. Many organizations are still behind the power curve when it comes to updating and security other apps like Adobe Reader. Count me among the camp that would like to see Microsoft opening its update mechanism to other vendors. It shouldn't be so difficult to keep the suite of third-party software on a typical Windows PC fully patched.
The other issue is that MS Office formats, particularly the older OLE style documents, are really complicated. While complexity typically provides fertile ground for potential vulnerabilities, finding ways to exploit those vulnerabilities can be difficult, meaning harder work for the attacker. Similarly, on the defensive side of the house, writing tools to look for dangerous payloads in MS Office apps are hampered by the same complexity issues. PDF seems simple when compared against MS Office files.
Lenny: Do you think programming or scripting skills are necessary for a security pro to succeed as a forensic analyst or incident responder? If one is going to learn a language, which one would you recommend?
Hal: There's the obvious point that if you're doing static analysis, having some programming background in algorithms and data structures is helpful in understanding the code you're analyzing. Basic knowledge of assembly conventions is nice, but I find knowledge of a slightly higher level language like C helps me formulate things more clearly as pseudo-code.
Also, the volume of attacks in a typical enterprise has increased well beyond the level where an organization can respond manually. Automation is required. Vendor tools are never going to address all of the problems you're going to face. So you better learn to write scripts.
We've been fielding a lot of questions on the Command-Line Kung Fu blog that come down to techniques for "finding evil"- in logs, file systems, memory images, incoming message streams, and so on. Command-line expertise on your chosen platform provides enormous leverage.
It also helps to learn a "real" scripting language for writing larger programs. I'm a Perl guy from way back, and I'm telling you that if you want to learn a scripting language, you're probably better off starting with Python. If you look at the popular projects out there, the vast majority are using Python. So I'll stick with my Perl, thank you very much, but you kids get off my lawn and start learning some Python.
Lenny: What are some of the non-technical skills that a forensic investigator or a malware analyst needs to have, if any?
Hal: Communication skills are number one. All the technical ability in the world to crack open malware isn't going to help if you're unable to communicate your findings to non-technical people. You also have to be able to effectively "manage up" and advocate for resources and funding, and for change in standard operating procedures when they no longer make sense.
The ability to organize and index large amounts of data is also becoming a real issue. Finding evil in a large enterprise network is a struggle to integrate Firewall, IDS, network capture, file system, registry, and application data into a complete story. You have to know where your organizational "crown jewels" are stashed. And you have to keep abreast of threat intelligence and track a growing collection of "bad actors" who are coming at your networks.
And you need to know how to research and learn on your own. Attacks are constantly evolving, and new vulnerabilities will always appear. Frankly, this is one of the reasons I love the field we're in- there's always a new technical challenge to attack.
Lenny: The recently-published details of the Koobface gang investigation showed that researchers were able to learn a lot about the IT infrastructure and people behind this malware campaign from the mistakes that the gang made. To what extent can a forensic analyst count on the adversary's mistakes when conducting an investigation?
Hal: This is very common in my experience. Consider that many attackers didn't start out their lives with the intention of infiltrating enterprise networks. So you'll find leads from their early career still floating around in the "Internet memory".
Also, try a simple thought exercise where you imagine trying to penetrate a typical PC and steal money. Think about all of the "moving pieces" of such an attack. Are you always going to use untraceable IP addresses to manage your malware distribution and command and control infrastructure or will you be in a hurry one day and use a personal IP address? How about clues in domain registrations, hosting agreements, or even the malware itself? How are you going to collect your money without leaving a trail that leads right back to you?
With persistence, a good analyst will always find something that enables them to get a bead on their adversary. Unfortunately, at the moment there are still too many "safe havens" in the world where the attackers can escape the consequences of their actions even after they've been identified.
Thanks for sharing your insights, Hal!
